Discussion 8

One of the most challenging things for students to understand is what it means to reject the null hypothesis. How is rejecting the null hypothesis related to statistical significance? Please use your own words (very important!) to describe what information you use to determine statistical significance, and give an example of this information (for example, how might you check to see if your oneway ANOVA is significant? How about a Two-Way ANOVA?). Once you decide that an effect is significant, for example, please indicate what that means to you IN YOUR OWN WORDS. The point of this discussion is to help you help each other learn, not to copy down a perfect answer. Finally, do you see any problems with/limitations of NHST? Why or why not?
—
In order to determine statistical significance, we need to decide whether the results of our experiment or study are due to chance or whether they are likely due to a real effect or relationship. To do this, we typically use a null hypothesis, which is a hypothesis that assumes there is no real effect or relationship. We then use statistical tests to see how likely it is that the results of our experiment or study could have occurred by chance if the null hypothesis were true. If the probability of getting our results by chance is very low (usually less than 5%), we can reject the null hypothesis and conclude that there is a statistically significant effect or relationship.

For example, if we were conducting a one-way ANOVA to compare the means of three different groups, we would start by assuming that there is no difference between the groups (the null hypothesis). We would then use the ANOVA test to calculate the probability of getting our observed results if the null hypothesis were true. If this probability is very low, we would reject the null hypothesis and conclude that there is a significant difference between the means of the groups.

Once we have determined that an effect is statistically significant, it means that it is very unlikely to have occurred by chance. This allows us to conclude that the effect is real and not just due to random variation.

One limitation of NHST (null hypothesis significance testing) is that it only allows us to determine whether an effect is statistically significant or not, but it does not tell us anything about the size or practical importance of the effect. For example, even a very small effect can be statistically significant if we have a large enough sample size, but it may not be practically meaningful. Therefore, it’s important to consider both statistical significance and the size of the effect when interpreting the results of a study.