HW 5 due Sunday at 11:59pm due to missing data file.
Lab 6 due this Friday 11:59 pm.
Overview
Comparing more than three means
ANOVA test and how it relates to a t-test
Multiple comparisons issue and Inflated type I error rates
Pulse example: ANOVA
Bonferroni Correction
Very optional reading
P&G: Chapter 12
OI: 7.5
Puppies
Experiment: Pets and Stress Level
For today’s topic, we will consider an experiment where heart rates were measured (as a proxy for stress level) for a group of \(n=90\) students after 10 minutes of being under certain conditions.
The 3 Conditions/Groups: Of the 90 total students, 30 students spent time holding puppies, 30 students sat alone, and 30 students chatted with a friend.
After 10 minutes, each student’s heart rate was measured.
Research Question: Is there a group (puppies, friend, alone/neither) associated with lower heart rate compared to the others?
Let’s take a look at the data on the next slide.
Distribution of heart rate data
library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.0 ✔ readr 2.1.6
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.1 ✔ tibble 3.3.0
✔ lubridate 1.9.4 ✔ tidyr 1.3.2
✔ purrr 1.2.0
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(ggplot2)# Set seed for reproducibilityset.seed(123)# Simulate data: 30 people per grouppuppies <-rnorm(30, mean=78, sd=7) # lower pulse for those with puppiesneither <-rnorm(30, mean=82, sd=5) # pulse for those alonefriends <-rnorm(30, mean=85, sd=5) # pulse for those chatting with a friend# Combine into a data framepulse_data <-data.frame(pulse =c(friends, neither, puppies),group =factor(rep(c("Friend", "Neither", "Puppies"), each=30)))ggplot(data = pulse_data, mapping =aes(x = group, y = pulse)) +geom_boxplot() +scale_colour_brewer(palette ="Set1")
How do we compare across three groups? Which groups are different?
Example: pets and stress
We are interested in testing
\[H_0: \mu_P = \mu_F = \mu_N\]
against the alternative that at least one mean is different from the others.
One way to do this would be to use t-tests on all possible pairs of tests (here there are just three - how many pairs of tests would this be?). However, if we have more groups, this becomes quite complicated.
For example, with 10 groups we need to do \(10\choose {2}\) = 45 tests!
Review: Error rates
Tip
In groups, fill in the blanks:
Suppose we test the null hypothesis \(H_0\):\(\mu = \mu_0\). We could potentially make two types of errors:
Truth
\(\mu = \mu_0\)
\(\mu \neq \mu_0\)
Fail to reject \(H_0\)
correct decision
____________
Reject \(H_0\)
___________
correct decision
How would you explain the following in words?
Type I Error: ____________
Type II Error: ____________
Multiple comparisons
Now that we’ve reviewed Type I and Type II errors, we need to discuss multiple comparisons - an important concept in statistics.
Back to our pet example. What if we had 10 groups?
In addition to being time-consuming, carrying out multiple tests can lead to inflated Type I error rates which puts into question the validity of a given study if these multiple comparisons are not accounted for.
Multiple comparisons in action: overall test
Extra tests…whoa!
Green jelly beans!
So what went wrong?
Multiple comparisons
Consider our pets / stress example, where we had three pairwise comparisons.
Suppose all means are truly equal and we conduct all three pairwise tests
\(H_0\): \(\mu_P = \mu_F\) vs. \(H_A\): \(\mu_P \neq \mu_F\)
\(H_0\): \(\mu_P = \mu_N\) vs. \(H_A\): \(\mu_P \neq \mu_N\)
\(H_0\): \(\mu_N = \mu_F\) vs. \(H_A\): \(\mu_N \neq \mu_F\)
Suppose also the tests are independent and done at a 0.05 significance level
Then the probability we fail to reject all three tests if the null hypothesis were true (i.e. the correct decision), assuming the tests are independent, is:
\(P(\text{fail to reject Test 1} | H_0 \text{ true})\cdot P(\text{fail to reject Test 2} | H_0 \text{ true}) \cdot P(\text{fail to reject Test 3} | H_0 \text{ true})\)
Multiple comparisons (continued)
Assuming the tests are independent, this probability is \((1−0.05)^3=0.95^3=0.857\) , and so the probability of rejecting at least one of the three null hypotheses, called the family-wise error rate, is \(1−0.857=0.143>0.05\)
If we had 10 groups instead, we would need 45 pairwise tests. With 45 tests, the probability of rejecting at least 1 of them (incorrectly!) is over 90%!
Each hypothesis test carries a small chance (e.g., 5%) of being a false positive.
Like flipping a coin: occasionally, you’ll get “heads” just by chance.
When we run many tests, those small chances add up.
With only 3 tests, there’s already a 14% chance of at least one false alarm. With 45 tests, that chance exceeds 90%!
In practice: Even if nothing real is going on, we’re almost guaranteed to find something “statistically significant.” This can lead to misleading conclusions — especially in large studies testing many variables.
Takeaway: The more tests we do, the more we must adjust for multiple comparisons (e.g., Bonferroni correction, FDR (false discovery) control) to avoid fooling ourselves.
Multiple comparisons
ANOVA extends the t-test and is one way to control the overall Type I error rate at a fixed level \(\alpha\), if we only test pairwise differences when the overall ANOVA test is rejected
ANOVA stands for analysis of variance. We use ANOVA when we want to compare more than two groups.
ANOVA null hypothesis
In ANOVA, we typically follow this testing procedure:
First, we conduct an overall test of the null hypothesis that the means of all of the groups are equal.
If this is rejected, then we step down to see which means are different from each other. A multiple comparisons correction is sometimes used for these pairwise comparisons of means.
If we fail to reject the null hypothesis, then no further testing should be done.
ANOVA alternative hypothesis
For ANOVA with three groups, our null hypothesis is \(H_0: \mu_P = \mu_F = \mu_N\). What could happen under the alternative?
\(μ_P≠μ_F≠μ_N\)
\(μ_P=μ_F\), but \(μ_N\) is different
\(μ_P=μ_N\), but \(μ_F\) is different
\(μ_F=μ_N\), but \(μ_P\) is different
The alternative hypothesis for ANOVA is that at least one of the means is different from the others.
Assumptions of ANOVA
Outcomes within groups are normally distributed
Homoscedastic variance (the within-group variance is the same for all groups)
Samples are independent
If these assumptions are violated, then results from ANOVA may not be valid. We will discuss some alternatives later in the course.
Variances appear to be similar, but normality may be slightly questionable. For now, let’s proceed despite this.
Why analyze variance when we want means?
Remember, ANOVA stands for analysis of variance.
What does variance have to do with our null hypothesis, which is about equality of means (say, \(H_0:μ_1=μ_2=⋯=μ_K\))?
We are analyzing (sources of) variance in the response variable \(Y\).
Why analyze variance when we want means?
If the means \(\mu_1, ..., \mu_K\) are different, that is one source of variance in \(Y\).
There is essentially always random variation as well, with variance \(\sigma^2\)
Analysis of variance attempts to determine whether any of the variance in \(Y\) is attributable to differences between groups, or whether it’s all random variation.
In essence, compare between-group variance (\(s^2_{between}\)) to within-group variance (\(s^2_{between}\))
F-test
How do we formally compare these variances? Consider the F statistic given by
\[F = \frac{n_k s^2_{between}}{s^2_{within}}\]
where \(n_k\) is the size of each group. If \(H_0\) is true, has an F distribution with K−1 numerator degrees of freedom and \(n−K\)denominator degrees of freedom, where K is the number of groups and \(n\) is the total number of observations.
F-test
The F-test for ANOVA is inherently one-tailed, rejecting \(H_0\) only if F is considerably larger than one.
Importantly, this does not mean we have a one-sided alternative; we just look at one tail of the F distribution to get the p-value.
If there are only two groups, then the F-test gives the same result as the independent-samples t-test.
Output
# anova in R:summary(aov(pulse ~ group, data = pulse_data))
Df Sum Sq Mean Sq F value Pr(>F)
group 2 877.7 438.8 15.77 1.44e-06 ***
Residuals 87 2421.7 27.8
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Which rows correspond to the between-group and within-group variances here, respectively?
F-test for pet data
Note that \(F = n_k s^2_B / s^2_W = \frac{(877.7/2)}{(2421.7/87)} = 15.77\), with ndf = \(3-1 = 2\) and ddf \(90-3=87\).
This corresponds to a p-value <.0001.
Conclusion: At \(\alpha = 0.05\), we reject the null hypothesis. There is sufficient evidence that at least one of the three groups comes from a population with a different mean than the others.
Which groups might be different?
Application Exercise
On Canvas, download today’s Application Exercise template.
The application Exercise will be due Monday at 11:59pm.
Bonferroni correction
As we showed earlier, conducting multiple tests on a data set increases the family-wise error rate.
One very conservative way to ensure this is not the case is to simply divide α by the number of tests to be done and to use that as the significance level.
This procedure is called the Bonferroni correction.
Bonferroni correction
For example: for two tests, to preserve an overall 0.05 type I error rate, the Bonferroni correction would use \(\alpha/2=0.025\) as the significance level for each individual test instead of 0.05.
Bonferroni is a conservative correction, making it harder to reject the null hypothesis, but it is a safe bet in controlling the Type I error rate.
Pets and stress: group differences
We can compare the groups using a Bonferroni correction (here we have three tests, so the significance level for each test is \(\alpha/3\)).
The raw (uncorrected) p-values for the t-test comparing friend vs. pet was \(<0.0001\); for friend vs. neither was 0.021, and for pet vs. neither was 0.009.
