Lecture 12: ANOVA

BIOS 600 - Spring 2026

Announcements

  • HW 5 due Sunday at 11:59pm due to missing data file.

  • Lab 6 due this Friday 11:59 pm.

Overview

  • Comparing more than three means

  • ANOVA test and how it relates to a t-test

  • Multiple comparisons issue and Inflated type I error rates

  • Pulse example: ANOVA

  • Bonferroni Correction

Very optional reading

  • P&G: Chapter 12

  • OI: 7.5

Puppies

Experiment: Pets and Stress Level

  • For today’s topic, we will consider an experiment where heart rates were measured (as a proxy for stress level) for a group of \(n=90\) students after 10 minutes of being under certain conditions.

  • The 3 Conditions/Groups: Of the 90 total students, 30 students spent time holding puppies, 30 students sat alone, and 30 students chatted with a friend.

  • After 10 minutes, each student’s heart rate was measured.

  • Research Question: Is there a group (puppies, friend, alone/neither) associated with lower heart rate compared to the others?

  • Let’s take a look at the data on the next slide.

Distribution of heart rate data

How do we compare across three groups? Which groups are different?

Example: pets and stress

We are interested in testing

\[H_0: \mu_P = \mu_F = \mu_N\]

against the alternative that at least one mean is different from the others.

  • One way to do this would be to use t-tests on all possible pairs of tests (here there are just three - how many pairs of tests would this be?). However, if we have more groups, this becomes quite complicated.

  • For example, with 10 groups we need to do \(10\choose {2}\) = 45 tests!

Review: Error rates

Tip

In groups, fill in the blanks:

  • Suppose we test the null hypothesis \(H_0\):\(\mu = \mu_0\). We could potentially make two types of errors:
Truth \(\mu = \mu_0\) \(\mu \neq \mu_0\)
Fail to reject \(H_0\) correct decision ____________
Reject \(H_0\) ___________ correct decision

How would you explain the following in words?

  • Type I Error: ____________
  • Type II Error: ____________

Multiple comparisons

  • Now that we’ve reviewed Type I and Type II errors, we need to discuss multiple comparisons - an important concept in statistics.

  • Back to our pet example. What if we had 10 groups?

  • In addition to being time-consuming, carrying out multiple tests can lead to inflated Type I error rates which puts into question the validity of a given study if these multiple comparisons are not accounted for.

Multiple comparisons in action: overall test

Extra tests…whoa!

Green jelly beans!

So what went wrong?

Multiple comparisons

Consider our pets / stress example, where we had three pairwise comparisons.

  • Suppose all means are truly equal and we conduct all three pairwise tests

  1. \(H_0\): \(\mu_P = \mu_F\) vs. \(H_A\): \(\mu_P \neq \mu_F\)

  2. \(H_0\): \(\mu_P = \mu_N\) vs. \(H_A\): \(\mu_P \neq \mu_N\)

  3. \(H_0\): \(\mu_N = \mu_F\) vs. \(H_A\): \(\mu_N \neq \mu_F\)

  • Suppose also the tests are independent and done at a 0.05 significance level

  • Then the probability we fail to reject all three tests if the null hypothesis were true (i.e. the correct decision), assuming the tests are independent, is:

\(P(\text{fail to reject Test 1} | H_0 \text{ true})\cdot P(\text{fail to reject Test 2} | H_0 \text{ true}) \cdot P(\text{fail to reject Test 3} | H_0 \text{ true})\)

Multiple comparisons (continued)

  • Assuming the tests are independent, this probability is \((1−0.05)^3=0.95^3=0.857\) , and so the probability of rejecting at least one of the three null hypotheses, called the family-wise error rate, is \(1−0.857=0.143>0.05\)

  • If we had 10 groups instead, we would need 45 pairwise tests. With 45 tests, the probability of rejecting at least 1 of them (incorrectly!) is over 90%!

Intuitive Takeaway: Why Multiple Comparisons Matter

  • Each hypothesis test carries a small chance (e.g., 5%) of being a false positive.

    • Like flipping a coin: occasionally, you’ll get “heads” just by chance.

  • When we run many tests, those small chances add up.

    • With only 3 tests, there’s already a 14% chance of at least one false alarm. With 45 tests, that chance exceeds 90%!

  • In practice: Even if nothing real is going on, we’re almost guaranteed to find something “statistically significant.” This can lead to misleading conclusions — especially in large studies testing many variables.

  • Takeaway: The more tests we do, the more we must adjust for multiple comparisons (e.g., Bonferroni correction, FDR (false discovery) control) to avoid fooling ourselves.

Multiple comparisons

  • ANOVA extends the t-test and is one way to control the overall Type I error rate at a fixed level \(\alpha\), if we only test pairwise differences when the overall ANOVA test is rejected

  • ANOVA stands for analysis of variance. We use ANOVA when we want to compare more than two groups.

ANOVA null hypothesis

In ANOVA, we typically follow this testing procedure:

  1. First, we conduct an overall test of the null hypothesis that the means of all of the groups are equal.

  2. If this is rejected, then we step down to see which means are different from each other. A multiple comparisons correction is sometimes used for these pairwise comparisons of means.

  3. If we fail to reject the null hypothesis, then no further testing should be done.

ANOVA alternative hypothesis

For ANOVA with three groups, our null hypothesis is \(H_0: \mu_P = \mu_F = \mu_N\). What could happen under the alternative?

  • \(μ_P≠μ_F≠μ_N\)

  • \(μ_P=μ_F\), but \(μ_N\) is different

  • \(μ_P=μ_N\), but \(μ_F\) is different

  • \(μ_F=μ_N\), but \(μ_P\) is different

The alternative hypothesis for ANOVA is that at least one of the means is different from the others.

Assumptions of ANOVA

  1. Outcomes within groups are normally distributed

  2. Homoscedastic variance (the within-group variance is the same for all groups)

  3. Samples are independent

If these assumptions are violated, then results from ANOVA may not be valid. We will discuss some alternatives later in the course.

Validity of ANOVA for pet data

pulse_data |>
  ggplot(mapping = aes(x = pulse)) + 
  geom_histogram(bins=10) +
  facet_wrap(~group)

Validity of ANOVA for pet data

Variances appear to be similar, but normality may be slightly questionable. For now, let’s proceed despite this.

Why analyze variance when we want means?

  • Remember, ANOVA stands for analysis of variance.

  • What does variance have to do with our null hypothesis, which is about equality of means (say, \(H_0:μ_1=μ_2=⋯=μ_K\))?

  • We are analyzing (sources of) variance in the response variable \(Y\).

Why analyze variance when we want means?

  • If the means \(\mu_1, ..., \mu_K\) are different, that is one source of variance in \(Y\).

  • There is essentially always random variation as well, with variance \(\sigma^2\)

  • Analysis of variance attempts to determine whether any of the variance in \(Y\) is attributable to differences between groups, or whether it’s all random variation.

  • In essence, compare between-group variance (\(s^2_{between}\)) to within-group variance (\(s^2_{between}\))

F-test

How do we formally compare these variances? Consider the F statistic given by

\[F = \frac{n_k s^2_{between}}{s^2_{within}}\]

where \(n_k\) is the size of each group. If \(H_0\) is true, has an F distribution with K−1 numerator degrees of freedom and \(n−K\) denominator degrees of freedom, where K is the number of groups and \(n\) is the total number of observations.

F-test

  • The F-test for ANOVA is inherently one-tailed, rejecting \(H_0\) only if F is considerably larger than one.

  • Importantly, this does not mean we have a one-sided alternative; we just look at one tail of the F distribution to get the p-value.

  • If there are only two groups, then the F-test gives the same result as the independent-samples t-test.

Output

# anova in R:
summary(aov(pulse ~ group, data = pulse_data))
            Df Sum Sq Mean Sq F value   Pr(>F)    
group        2  877.7   438.8   15.77 1.44e-06 ***
Residuals   87 2421.7    27.8                     
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Which rows correspond to the between-group and within-group variances here, respectively?

F-test for pet data

  • Note that \(F = n_k s^2_B / s^2_W = \frac{(877.7/2)}{(2421.7/87)} = 15.77\), with ndf = \(3-1 = 2\) and ddf \(90-3=87\).

  • This corresponds to a p-value <.0001.

  • Conclusion: At \(\alpha = 0.05\), we reject the null hypothesis. There is sufficient evidence that at least one of the three groups comes from a population with a different mean than the others.

  • Which groups might be different?

Application Exercise

  • On Canvas, download today’s Application Exercise template.

  • The application Exercise will be due Monday at 11:59pm.

Bonferroni correction

  • As we showed earlier, conducting multiple tests on a data set increases the family-wise error rate.

  • One very conservative way to ensure this is not the case is to simply divide α by the number of tests to be done and to use that as the significance level.

  • This procedure is called the Bonferroni correction.

Bonferroni correction

  • For example: for two tests, to preserve an overall 0.05 type I error rate, the Bonferroni correction would use \(\alpha/2=0.025\) as the significance level for each individual test instead of 0.05.

  • Bonferroni is a conservative correction, making it harder to reject the null hypothesis, but it is a safe bet in controlling the Type I error rate.

Pets and stress: group differences

  • We can compare the groups using a Bonferroni correction (here we have three tests, so the significance level for each test is \(\alpha/3\)).

  • The raw (uncorrected) p-values for the t-test comparing friend vs. pet was \(<0.0001\); for friend vs. neither was 0.021, and for pet vs. neither was 0.009.

In your small groups: What might we conclude?