3.8 Inference for Experiments

3.8 Inference for Experiments

Learning Target

• Outline an experiment that uses a completely randomized design.

Learning Target

• Explain the concept of statistical significance in the context of an experiment.

Statistical Significance

When an observed difference in responses between the groups in an experiment is too large to be explained by chance variation in the random assignment, we say that the result is statistically significant.  

Example

• The mean improvement for patients with the active magnet is 5.24 and the mean improvement for the patients with inactive magnets is 1.10, resulting in a difference of 5.24 - 1.10 = 4.14.

• The researchers in this study determined that the difference of 4.14 was statistically significant. In other words, a difference in mean improvement of 4.14 was too large to occur simply due to chance variability in the random assignment.

Example

• What if the researchers in this study determined that the difference of 4.14 was not statistically significant.

• A difference in mean improvement of 4.14 was small enough that it could be due to chance variation in the random assignment to treatments.

Multiple Choice

Statistically significant means

• a difference in mean too large to occur simply due to chance variability in the random assignment.

• a difference in mean small enough that it could be due to chance variation in the random assignment.

Multiple Choice

Not statistically significant means

• a difference in mean too large to occur simply due to chance variability in the random assignment.

• a difference in mean small enough that it could be due to chance variation in the random assignment.

Learning Target

• Use simulation to determine if the difference between two means or two proportions in an experiment is significant.

Statistical Significance with Simulation

Because the difference of 1.2 or greater is somewhat likely to occur by chance alone, the results of the experiment aren’t statistically significant.

Statistically significant

For now, we recommend using a boundary of 5% so that differences that would occur less than 5% of the time by chance alone are considered statistically significant. 

Multiple Choice

Is a difference of 0.2 statistically significant?

(100 dots total)

• No

• Yes

Multiple Choice

Is a difference of 0.4 statistically significant?

(100 dots total)

• No

• Yes

Multiple Choice

Is a difference of 0.3 statistically significant?

(100 dots total)

• No

• Yes