Probability | |
Distance from expected value (in standard errors) |
We can see, for instance, that, if the observed value is 2 standard errors from the expected value, the probability of getting a value at least that different if the hypothesis is true is about 0.05, or 5%.
A critical value is a value determined in advance to decide whether a hypothesis will be accepted or rejected. If an observed value is at or beyond the critical value (in the rejection region), the hypothesis is rejected; otherwise (if the observed value is in the acceptance region), the hypothesis is accepted. In the chart above, if the critical value had been set at 2 standard errors, there would be about a 5% chance, if the hypothesis were true, that it would be rejected.
If the hypothesis is the null hypothesis, rejecting it when it is true is called a type-I error. If the hypothesis in the chart is the null hypothesis, the probability of a type-I error if the critical value is set at 2 standard errors is about 5%.
Accepting the null hypothesis when it is false is called a type-II error. The probability of a type-II error is hard to estimate, because there are so many different ways in which a hypothesis could be false. Generally, however, the probability of a type-II error is less for larger samples than for smaller samples.