Statwing's approach to statistical testing

# Statistical Significance (T-Test) Statwing represents t-test results as distribution curves.
Assuming there is a large enough sample size, the difference between these two groups is probably statistically significant.

Note: The below discusses statistical significance for the unranked “Independent Samples T-Test”, the most common form of T-Test.

## Definition

A t-test’s statistical significance indicates whether or not the difference between two groups’ averages most likely reflects a “real” difference in the population from which the groups were sampled.

## Example

Let’s say you’re interested in whether the average New Yorker spends more than the average Kansan per month on movies.

You ask a sample of 3 people from each state about their movie spending. You might observe a difference in those averages (like \$14 for the average Kansan and \$18 for the average New Yorker). But that difference is not statistically significant; it could easily just be random luck of which 3 people you randomly sampled that makes one group appear to spend more money than the other. If instead you ask 300 New Yorkers and 300 Kansans and still see a big difference, that difference is less likely to be caused by the sample being unrepresentative.

Note that if you asked 300,000 New Yorkers and 300,000 Kansans, the result would likely be statistically significant even if the difference between the group was only a penny. The t-test’s effect size complements its statistical significance, describing the magnitude of the difference, whether or not the difference is statistically significant.

## Definition

A statistically significant t-test result is one in which a difference between two groups is unlikely to have occurred because the sample happened to be atypical. Statistical significance is determined by the size of the difference between the group averages, the sample size, and the standard deviations of the groups. For practical purposes statistical significance suggests that the two larger populations from which we sample are “actually” different.