4  Hypothesis tests

We have up to now discussed the properties of estimators that allow us to characterize their distributions in finite and large samples. These properties allow us to say, for example, that our estimated difference in means is equal to a true average treatment effect on average across repeated samples or that it will converge to the true value in large samples. These properties, however, are properties of repeated samples. Most researchers, on the other hand, will only have access to a single sample. Statistical inference is the process of using a single sample to learn about population parameters. As we will see, many common techniques of statistical inference are intuitively closely connected. One of the most ubiquitous in the social sciences is the hypothesis test, a kind of statistical thought experiment.

Example 4.1 (The Lady Tasting Tea) The story of the Lady Tasting Tea exemplifies the core ideas behind hypothesis testing.¹ The story goes like this. R. A. Fisher, the early 20th-century British polymath and statistical pioneer, poured a cup of tea for his colleague, the algologist Muriel Bristol, who rejected the cup, stating that she preferred pouring the milk into the cup first, then the tea. Fisher was skeptical of the idea that anyone could tell the difference between a cup poured milk-first versus tea-first, and so he and another colleague, William Roach, devised a test to see if Bristol could tell the difference between the two preparation methods.

Fisher and Roach prepared 8 cups of tea, four with the milk poured first and four with the tea poured first. Then they presented the cups to Bristol in a random order (though she knew there were four of each type), and she reportedly proceeded to identify all of the cups correctly. At first glance, this seems like good evidence that she could tell the difference between the two types of tea, but Fisher, being a natural skeptic, raised the question, “Could she have just been randomly guessing and gotten lucky?” This led Fisher to a statistical thought experiment: what would the probability of identifying the correct cups be if she had no ability to discern?

To calculate the probability of Bristol identifying the four milk-first cups correctly, note that having no ability to discern would mean that the number of correct selections would be determined by Fisher and Roach’s random selection of 4 cups to be prepared milk-first from the 8 cups available. Using basic combinatorics, there are 70 ways to choose 4 cups among 8, but only 1 of those arrangements would perfectly align with her selections. Thus, the probability of guessing the right set of cups is 1/70 or \(\approx 0.014\). The low probability implies that the hypothesis of “no milk-first discernment” may be implausible.

The story of the Lady Tasting Tea encapsulates many of the core elements of hypothesis testing. Hypothesis testing is about taking our observed estimate (Bristol identifying all four cups correctly) and seeing how likely an estimate at least as extreme as the observed estimate would be under some assumption, or hypothesis, about the data-generating process (Bristol had no ability to discern). When the observed estimate is unlikely under the maintained hypothesis, we might view this as evidence against that hypothesis. Thus, hypothesis tests help us assess evidence for particular guesses about the DGP.

This chapter proceeds as follows. First, we will introduce the structure of hypothesis tests—the notation, concepts, and key design choices. Then we will work through a common, specific test of the sample mean. We then explore a broadly applicable and commonly used test for asymptotically normal estimators called the Wald test. We then discuss \(p\)-values and power analyses before turning to two types of exact tests that do not rely on asymptotic approximations. Finally, we end the chapter with a discussion of multiple testing and ways to address it.

NoteNotation alert

For most of this chapter, we will introduce the concepts following the notation in the past chapters except where noted. We will assume a random (iid) sample of random variables \(X_1, \ldots, X_n\) from a distribution, \(F\). We’ll focus on estimating some parameter, \(\theta\), of this distribution (like the mean, median, variance, etc.), and we will refer to \(\Theta\) as the set of possible values of \(\theta\) or the parameter space.

4.1 Structure of hypothesis tests

4.1.1 Hypotheses

In the context of hypothesis testing, hypotheses are simply statements about the population distribution. In particular, we will make statements that \(\theta = \theta_0\) where \(\theta_0 \in \Theta\) is the hypothesized value of \(\theta\), a population parameter. Hypotheses are ubiquitous in empirical work. Examples include:

• The population proportion of US citizens who identify as Democrats is 0.33.
• The population difference in average voter turnout between households who received get-out-the-vote mailers vs. those who did not is 0.
• The difference in the average incidence of human rights abuse in countries that signed a human rights treaty vs. those countries that did not sign is 0.

Each of these is a statement about the true DGP. The latter two are examples where the hypothesis is phrased as a possible non-difference, which is very common. When \(\theta\) represents the difference in means between two groups, then \(\theta = 0\) is the hypothesis of no actual difference in population means or no treatment effect (if the causal effect is identified).

The goal of hypothesis testing is to adjudicate between two complementary hypotheses.

Definition 4.1 The two hypotheses in a hypothesis test are called the null hypothesis and the alternative hypothesis, denoted as \(H_0\) and \(H_1\), respectively.

These hypotheses are complementary, so if the null hypothesis is \(H_0: \theta \in \Theta_0\), then the alternative hypothesis is \(H_1: \theta \in \Theta_0^c\). The “null” in null hypothesis may seem odd until you realize that most null hypotheses are that there is no effect of some treatment or no difference in means. For example, suppose that \(\theta\) is the difference in mean support for increasing legal immigration between a treatment group that received a pro-immigrant message with some facts about immigration and a control group that just received the immigration facts. The usual null hypothesis would be no difference in means or \(H_0: \theta = 0\), and the alternative would be \(H_1: \theta \neq 0\). Substantively, the null hypothesis would posit no average difference in the outcome—in this case support for increasing legal immigration—between the two groups.

There are two common types of tests that differ in terms of the form of their null and alternative hypotheses. A two-sided test is of the form \[ H_0: \theta = \theta_0 \quad\text{versus}\quad H_1: \theta \neq \theta_0, \] where the “two-sided” part refers to how the alternative contains values of \(\theta\) above and below the null value \(\theta_0\).

A one-sided test is of the form \[ H_0: \theta \leq \theta_0 \quad\text{versus}\quad H_1: \theta > \theta_0, \] or \[ H_0: \theta \geq \theta_0 \quad\text{versus}\quad H_1: \theta < \theta_0, \] where the “one-sided” part refers to how the alternative contains values of \(\theta\) only above or below the null value. As a way to generically refer to a null hypothesis without specifying whether it is one- or two-sided, we often write the null as \(\theta \in \Theta_0\).

Example 4.2 Suppose that \((X_1, \ldots, X_n)\) represents a sample of US citizens where \(X_i = 1\) indicates support for the current US president and \(X_i = 0\) means opposition (no support). We may want to know if there is evidence for the president having more than a bare majority support of the public. To do so, we can set the null hypothesis to be the opposite claim—that the president does not have the support of a majority of American citizens. Let \(\mu = \E[X_i] = \P(X_i = 1)\) be the population proportion supporting the president. Then, a one-sided test would compare the two hypotheses: \[ H_0: \mu \leq 0.5 \quad\text{versus}\quad H_1: \mu > 0.5. \]

Two-sided tests are much more common in the social sciences, mostly because we usually want to know if there is any evidence, positive or negative, against the presumption of no treatment effect or no relationship between two variables. One-sided tests are best suited for situations with clear, directional hypotheses that are ideally preregistered before collection of the data. Preregistration of the direction of a one-sided test is important because researchers changing the direction of the hypothesis after seeing the data can inflate the strength of evidence against the null. For this reason, one-sided tests outside of preregistered settings should be used with extreme caution.

4.1.2 The procedure of hypothesis testing

At the most basic level, a hypothesis test is a rule that specifies values of the sample data for which we will decide to reject the null hypothesis. Let \(\mathcal{X}_n\) be the support of the sample—that is, the set of all possible vectors \((x_1, \ldots, x_n)\). A hypothesis test then describes a region of this space, \(R \subset \mathcal{X}_n\), called the rejection region, such that when \((X_1, \ldots, X_n) \in R\) we will reject \(H_0\) and when the data is outside this region, \((X_1, \ldots, X_n) \notin R\), we retain, accept, or fail to reject the null hypothesis.²

How do we decide what the rejection region should be? Even though we define the rejection region in terms of the support, \(\mathcal{X}_n\), working with the entire vector of data can be unwieldy. We instead usually formulate the rejection region in terms of a test statistic, \(T = T(X_1, \ldots, X_n)\), which is a function of the data that should get larger as the observed data becomes incompatible with the null hypothesis. The test statistic is a random variable and has a distribution, though we usually design it so that, at least in large samples, its distribution under the null does not depend on unknown features of the data-generating process.

Once we have selected a test statistic, we can define the rejection region as \[ R = \left\{(x_1, \ldots, x_n) : T(x_1, \ldots, x_n) \geq c\right\}, \] where \(c\) is called the critical value. This expression says that the rejection region is the collection of possible data sets that make the test statistic sufficiently large. The critical value (and thus the rejection region) demarcates when the divergence between the observed data and the null hypothesis is large enough to allow us to reject the null hypothesis.

Consider a simple one-sided test where you feel a bit ill and try to determine if you have a normal body temperature of 98.6 degrees Fahrenheit or if you have a fever. In this case, the thermometer reading is the test statistic since larger readings are less consistent with a normal body temperature. Thermometers, however, are imperfect and noisy tools, so the reading might differ from 98.6 even if one’s temperature is normal. Thus, we can use a rejection region such as readings over 100.5 degrees to determine when to reject the null hypothesis of a normal body temperature.

The structure of a rejection region will depend on whether a test is one- or two-sided. This is an important point of difference between the two test types that we will raise again below. One-sided tests will take the form \(T \geq c\), whereas two-sided tests will take the form \(|T| \geq c\) since we want to count deviations from either side of the null hypothesis as evidence against that null.

4.1.3 Testing errors

Hypothesis tests end with a decision to reject the null hypothesis or not, but this might be an incorrect decision. In particular, there are two ways to make errors and two ways to be correct in this setting, as shown in Table 4.1. The labels are confusing, but remember that Type I errors (said “type one”) are false positives, which we usually consider the more problematic of the two types of errors. Type I errors occur when we reject a null when the null is in fact true. For example, if we have a true null hypothesis of no treatment effect, and we reject that null hypothesis (and conclude substantively that there is some sort of a treatment effect), then we would be committing a Type I error. Type I errors are what we see in the replication crisis: lots of “significant” effects that turn out later to be null.

Type II errors (said “type two”) are false negatives and are generally considered less problematic. For such errors, there is a true relationship, but we cannot detect it with our test. That is, we do not reject a null that is false. For example, if we have a null hypothesis of no treatment effect between a treatment and control condition, we would be committing a Type II error if in fact there was a difference in the treatment and control but we concluded there wasn’t (we failed to reject the null hypothesis of no difference).

Table 4.1: Typology of testing errors

	\(H_0\) True	\(H_0\) False
Retain \(H_0\)	Awesome	Type II error
Reject \(H_0\)	Type I error	Great

Ideally, we would minimize the chances of making either a Type I or Type II error. Unfortunately, because the test statistic is a random variable, we cannot remove the probability of an error altogether. Instead, we will derive tests that can cap the probability of Type I error, usually the more objectionable type of error. To derive this, we can define the power function of a test, \[ \pi(\theta) = \P\left( \text{Reject } H_0 \mid \theta \right), \] which is the probability of rejection as a function of the parameter of interest, \(\theta\). Here, \(\P(A \mid \theta)\) is the probability of \(A\) as if we knew the true parameter were \(\theta\). The power function tells us, for example, how likely we are to reject the null hypothesis of no treatment effect (no difference) as we vary the actual size of the treatment effect (which in this case is \(\theta\)).

We can define the probability of Type I error from the power function.

Definition 4.2 The size of a hypothesis test with the null hypothesis \(H_0: \theta \in \Theta_0\) is \[ \alpha = \sup_{\theta_0 \in \Theta_0} \pi(\theta_0) = \sup_{\theta_0 \in \Theta_0} \P\left( \text{Reject } H_0 \mid \theta_0 \right). \]

The size of the test is the largest probability of rejecting when the true parameter is in the null hypothesis region, or the false positive rate among true nulls.

Figure 4.1 shows an example of rejection regions, size, and power for a one-sided test with null hypothesis \(H_0: \theta \leq \theta_0\). In the left panel, we have the distribution of the test statistic under the boundary null hypothesis, \(\theta = \theta_0\), and the rejection region is defined by values \(T \geq c\). Why choose the boundary point for the null hypothesis here? The probability of landing in the rejection region varies with the true value of the parameter in the null region, \(\theta \leq \theta_0\), and the size of the test is the largest of these probabilities. In this case, \(\pi(\theta)\) is increasing in \(\theta\), so the rejection probability among the null values will be the largest at the boundary \(\theta_0\).

We refer to the distribution of the test statistic under the null hypothesis as the null distribution or the reference distribution. The shaded gray region is the probability of rejection under this null hypothesis, or the size of the test. Sometimes, we will get extreme samples by random chance, even under the null, leading to false discoveries.

In the right panel, we overlay the distribution of the test statistic under one particular alternative, \(\theta = \theta_1 > \theta_0\). The red-shaded region is the probability of rejecting the null when this alternative is true, or the power—it is the probability of correctly rejecting the null when it is false. Intuitively, we can see that alternatives that produce test statistics closer to the rejection region will have higher power. This makes sense: detecting big deviations from the null should be easier than detecting minor ones.

Figure 4.1: Size of a test and power against an alternative.

Figure 4.1 also hints at a tradeoff between size and power. Notice that we could make the size smaller (lower the false positive rate) by increasing the critical value to \(c' > c\). This would make the probability of being in the rejection region smaller, \(\P(T \geq c' \mid \theta_0) < \P(T \geq c \mid \theta_0)\), leading to a lower-sized test. Unfortunately, it would also reduce power in the right panel since the probability of being in the rejection region will be lower under any alternative, \(\P(T \geq c' \mid \theta_1) < \P(T \geq c \mid \theta_1)\). This means we usually cannot simultaneously reduce both types of errors, holding \(n\) fixed.

4.1.4 Determining the rejection region

If we cannot simultaneously optimize a test’s size and power, how should we determine where the rejection region is? That is, how should we decide what empirical evidence will be strong enough for us to reject the null? The standard approach is to control the size of a test (that is, control the rate of false positives) and try to maximize the power of the test subject to that constraint. So we say, “I’m willing to reject a true null at most X% of the time and do whatever we can to maximize power subject to that constraint.”

Definition 4.3 A test has significance level \(\alpha\) if its size is less than or equal to \(\alpha\), that is, \(\sup_{\theta_0 \in \Theta_0} \pi(\theta_0) \leq \alpha\).

A test with a significance level of \(\alpha = 0.05\) will have a false positive/Type I error rate no larger than 0.05. This level is widespread in the social sciences, though you also will see \(\alpha = 0.01\) or \(\alpha = 0.1\). Frequentists justify this by saying this means that with \(\alpha = 0.05\), at most 5% of studies in which the null is true will produce false positives.

Our task is to construct the rejection region so that the null distribution of the test statistic \(G_0(t) = \P(T \leq t \mid \theta_0)\) has no more than \(\alpha\) probability in that region. One-sided tests like the one in Figure 4.1 are the easiest to show, in spite of their problems discussed above. We want to choose \(c\) that puts no more than \(\alpha\) probability in the tail, or \[ \P(T \geq c \mid \theta_0) = 1 - G_0(c) \leq \alpha. \] Remember that using the smallest possible value of \(c\) maximizes power, which implies that the critical value for the maximum power while maintaining the significance level is when \(1 - G_0(c) = \alpha\). We can use the quantile function of the null distribution to find the exact value of \(c\) we need, \[ c = G^{-1}_0(1 - \alpha), \] which substantively translates to say, “the value below which \(1-\alpha\) of the null distribution falls.”

The determination of the rejection region follows the same principles for two-sided tests, but it is more complicated because we reject when the magnitude of the test statistic is large, \(|T| \geq c\). Figure 4.2 shows that basic setup. Notice that because there are two (disjoint) regions, one on the left and one on the right, we can write the size (false positive rate) as \[ \pi(\theta_0) = G_0(-c) + 1 - G_0(c). \] In most cases, the null distribution for such a test will be symmetric around 0 (usually asymptotically standard normal, actually), which means that \(G_0(-c) = 1 - G_0(c)\). This in turn implies that the size is \[ \pi(\theta_0) = 2(1 - G_0(c)). \] Solving for the critical value that would make this \(\alpha\) gives \[ c = G^{-1}_0(1 - \alpha/2). \] Again, this formula can seem dense, but remember what you are doing: finding the value that puts \(\alpha/2\) of the probability of the null distribution in each tail.

Figure 4.2: Rejection regions for a two-sided test along with the distribution of T under the null.

4.2 Hypothesis tests of the sample mean

Consider the following extended example about hypothesis testing of a sample mean, sometimes called a one-sample test since we are comparing a single sample mean to some sort of natural conceptual benchmark. Let’s say \(X_i\) represents feeling thermometer scores about “liberals” as a group on a scale of 0 to 100, with values closer to 0 indicating cooler feelings about liberals and values closer to 100 indicating warmer feelings about liberals. (This is similar to many survey items on nationally representative surveys, such as the ANES in the U.S.) We want to know if the population average differs from a value of 50, which is a good benchmark that would indicate roughly neutral feelings toward liberals. Suppose we obtained a sample mean of \(\Xbar_n = 53.1\) with a sample SD of \(S_n = 24\) on a sample size of \(n = 900\).

We can write this two-sided test as \[ H_0: \mu = 50 \quad\text{versus}\quad H_1: \mu \neq 50, \] where \(\mu = \E[X_i]\). The standard test statistic for this type of test is the so-called t-statistic, \[ T = \frac{\left( \Xbar_n - \mu_0 \right)}{\sqrt{S_n^2 / n}} =\frac{\left( \Xbar_n - 50 \right)}{\sqrt{S_n^2 / n}}, \] where \(\mu_0\) is the null value of interest and \(S_n^2\) is the sample variance. With the above observed data, we have \[ T = \frac{53.1 - 50}{24 / 30} = 3.875. \]

If the null hypothesis is true, then by the CLT and Slutsky’s Theorem from Chapter 3, we know that the t-statistic is asymptotically normal, \(T \indist \N(0, 1)\). Thus, we can approximate the null distribution with the standard normal.

Warning

The names of the various tests can be quite confusing because they are so similar. Earlier, we discussed one-sided versus two-sided tests, which differed in what alternative hypotheses were being considered. One-sample and two-sample tests, on the other hand, describe how many group means we are comparing. In a one-sample test, we compare one population mean to a fixed number. For two-sample tests (described in more detail below), we are usually making null hypotheses about the difference between two population means.

Let’s create a two-sided test with level \(\alpha = 0.05\), our tolerance for Type I error. Then we need to find the rejection region that puts \(0.05\) probability in the tails of the null distribution, which we just saw was \(\N(0,1)\). Let \(\Phi(\cdot)\) be the CDF for the standard normal and let \(\Phi^{-1}(\cdot)\) be the quantile function for the standard normal. Drawing on what we developed above, you can find the value \(c\) so that \(\P(|T| \geq c \mid \mu_0)\) is 0.05 with \[ c = \Phi^{-1}(1 - 0.05/2) \approx 1.96. \] This means that a test where we reject when \(|T| \geq 1.96\) would have a level of 0.05 asymptotically. In our example, \(|T| = 3.875 \geq 1.96\) so we reject the null.

4.3 The Wald test

We can generalize the hypothesis test for the sample mean to estimators more broadly.

Suppose \(\widehat{\theta}_n\) is an asymptotically normal estimator and the asymptotic variance of \(\sqrt{n}(\widehat{\theta}_n - \theta_0)\) is \(V_{\theta}\) and suppose we have a consistent estimator for that asymptotic variance, \(\widehat{V}_{\theta} \inprob V_{\theta}\). As in Chapter 3, we can define the estimated standard error as \(\widehat{\textsf{se}}[\widehat{\theta}_n] = \sqrt{\widehat{V}_{\theta} / n}\). We consider the two-sided test \[ H_0: \theta = \theta_0 \quad\text{versus}\quad H_1: \theta \neq \theta_0. \]

In many cases, our estimators will be asymptotically normal by a version of the CLT so that under the null hypothesis, we have \[ T = \frac{\widehat{\theta}_n - \theta_0}{\widehat{\textsf{se}}[\widehat{\theta}_n]} \indist \N(0, 1). \] The Wald test rejects \(H_0\) when \(|T| \geq z_{1-\alpha/2}\), where \(z_{a}\) is the \(a\)-th quantile of the standard normal. That is, if \(Z \sim \N(0, 1)\), then \(z_a\) satisfies \(\P(Z \leq z_a) = a\). While the \(1-\alpha/2\) expression is slightly confusing, the critical value \(z_{1-\alpha/2}\) is the value that puts probability \(\alpha/2\) in the upper tail of the standard normal.

Note

In R, you can find the \(z_{1- \alpha/2}\) values easily with the qnorm() function:

qnorm(1 - 0.05 / 2)

[1] 1.959964

Theorem 4.1 Suppose \(T\) is the test statistic for the Wald test for an asymptotically normal estimator with a consistent variance estimator. Then, the Wald test has asymptotic size \(\alpha\) such that \[ \P(|T| \geq z_{1 - \alpha/2} \mid \theta_0) \to \alpha. \]

This result is very general, and it means that many, many hypothesis tests based on estimators will have the same form. The main difference across estimators will be how we calculate the estimated standard error.

Example 4.3 (Difference in proportions) Get-out-the-vote (GOTV) experiments are common in political science. A typical GOTV design might randomly assign a group of citizens to receive mailers encouraging them to vote, whereas a control group receives no message. As in Section 2.8, we can define the data for each respondent as \((Y_i, D_i)\), where \(Y_i\) is a binary turnout variable (\(Y_i = 1\) if the respondent voted) and \(D_i\) is a binary treatment indicator (\(D_i = 1\) for respondents in the treated group). Let \(n_1 = \sum_{i=1}^n D_i\) be the number of treated respondents and \(n_0\) be the number of control respondents. Conditional on being in the treated or control group, we assume that the \(Y_i\) are iid draws from a Bernoulli distribution with success probabilities, \(\mu_1\) and \(\mu_0\), respectively (here the zero subscript indexes control rather than the null). That is, \(\mu_1\) is the probability of a treated respondent voting.

Our goal is to learn about the effect of this treatment on whether a citizen votes, \(\tau = \mu_1 - \mu_0\), and we will use the sample difference in means/proportions as our estimator, \(\widehat{\tau} = \Ybar_1 - \Ybar_0\). To perform a Wald test, we need to either know or estimate the standard error of this estimator. Recall from Section 2.8 that we know the conditional variance of our estimator is \[ \V[\widehat{\tau} \mid \mathbf{D}] = \frac{\V[Y_i \mid D_i = 1]}{n_1} + \frac{\V[Y_i \mid D_i = 0]}{n_0} = \frac{\mu_1(1-\mu_1)}{n_1} + \frac{\mu_0(1-\mu_0)}{n_0}, \] where the second equality comes from \(Y_i\) being conditionally Bernoulli, so \(\V[Y_i \mid D_i = 1] = \mu_1(1- \mu_1)\). Obviously, we do not know the true population proportions \(\mu_1\) and \(\mu_0\) (that’s why we’re doing the test!), but we can estimate the standard error by replacing them with their estimates \[ \widehat{\textsf{se}}[\widehat{\tau}] = \sqrt{\frac{\Ybar_1(1 -\Ybar_1)}{n_1} + \frac{\Ybar_0(1-\Ybar_0)}{n_0}}. \]

The typical null hypothesis test in this two-sample test is “no treatment effect” vs. “some treatment effect”: \[ H_0: \tau = \mu_1 - \mu_0 = 0 \quad\text{versus}\quad H_1: \tau \neq 0, \] which gives the following test statistic for the Wald test \[ T = \frac{\Ybar_1 - \Ybar_0}{\sqrt{\frac{\Ybar_1(1 -\Ybar_1)}{n_1} + \frac{\Ybar_0(1-\Ybar_0)}{n_0}}}. \] If we wanted a test with level \(\alpha = 0.01\), we would reject the null when \(|T| \geq 2.58\). We can find this critical value with the following R snippet:

qnorm(0.01 / 2, lower.tail = FALSE)

[1] 2.575829

Example 4.4 (Difference in means) Consider a similar example with randomly assigned treatment and control groups, but instead the treatment is now an appeal for financial donations to a political campaign and the outcomes are continuous measures of how much money a person has donated. The setup is the same as Section 2.8. The treatment data are iid draws from a population with mean \(\mu_1 = \E[Y_i \mid D_i = 1]\) and population variance \(\sigma^2_1 = \V[Y_i \mid D_i = 1]\). The control data are iid draws from a population with mean \(\mu_0 = \E[Y_i \mid D_i = 0]\) and population variance \(\sigma^2_0 = \V[Y_i \mid D_i = 0]\). The parameter of interest is similar to before: the population difference in means, \(\tau = \mu_1 - \mu_0\). We will form the usual hypothesis test of \[ H_0: \tau = \mu_1 - \mu_0 = 0 \quad\text{versus}\quad H_1: \tau \neq 0. \]

The only difference between this setting and the difference-in-proportions setting is that the standard error here is different because we cannot rely on binary outcomes. Instead, we’ll use our knowledge of the sampling variance of the sample means and independence between the samples to derive \[ \V[\widehat{\tau} \mid \mathbf{D}] = \V[\Ybar_1 \mid \mathbf{D}] + \V[\Ybar_0 \mid \mathbf{D}] = \frac{\sigma^2_1}{n_1} + \frac{\sigma^2_0}{n_0}, \] where we can come up with estimates of the unknown population variances with sample variances \[ \widehat{\se}[\widehat{\tau}] = \sqrt{\frac{S^2_1}{n_1} + \frac{S^2_0}{n_0}}. \] We can use this estimator to derive the Wald test statistic of \[ T = \frac{\widehat{\tau} - 0}{\widehat{\se}[\widehat{\tau}]} = \frac{\Ybar_1 - \Ybar_0}{\sqrt{\frac{S^2_1}{n_1} + \frac{S^2_0}{n_0}}}, \] and if we want an asymptotic level of 0.05, we can reject when \(|T| \geq 1.96\).

4.4 \(p\)-values

The hypothesis testing framework focuses on making a decision—to reject the null hypothesis or not—in the face of uncertainty. You choose a level of wrongness you are comfortable with (rate of false positives, or \(\alpha\)) and then decide null vs. alternative based firmly on the rejection region.

That said, note that we are discarding, somewhat artificially, information on how far the observed data is from the null hypothesis. For \(\alpha = 0.05\), we would “accept” the null if \(T = 1.95\) but would reject it if \(T = 1.97\), even though these are very similar. Simply reporting the reject/retain decision also fails to give us a sense of possible other levels at which we might have rejected the null. Again, this makes sense if we need to make a single decision: other tests don’t matter because we carefully considered our \(\alpha\) level test. But in the lower-stakes world of the academic social sciences, we can afford to be more informative.

One alternative to reporting the reject/retain decision is to report a \(p\)-value.

Definition 4.4 The \(p\)-value of a test is the probability of observing a test statistic at least as extreme as the observed test statistic, \(T_{\textsf{obs}}\), in the direction of the alternative hypothesis, assuming the null hypothesis is true.

The line “in the direction of the alternative hypothesis” deals with the unfortunate headache of one-sided versus two-sided tests. For a one-sided test where larger values of \(T\) correspond to more evidence for \(H_1\), the \(p\)-value for continuous test statistics is \[ \P(T(X_1,\ldots,X_n) \geq T_{\textsf{obs}} \mid \theta_0) = 1 - G_0(T_{\textsf{obs}}), \] whereas for a (symmetric) two-sided test, we have \[ \P(|T(X_1, \ldots, X_n)| \geq |T_{\textsf{obs}}| \mid \theta_0) = 2(1 - G_0(|T_{\textsf{obs}}|)). \]

In either case, the interpretation of the \(p\)-value is the same. It is the smallest level \(\alpha\) at which a test would reject the null hypothesis. Presenting a \(p\)-value allows the reader to determine their own \(\alpha\) level and determine quickly if the evidence would warrant rejecting \(H_0\) in that case. Thus, the \(p\)-value is a more continuous measure of divergence between the observed data and the null hypothesis.

Much of the controversy surrounding \(p\)-values (see Wasserstein and Lazar (2016) for a lively discussion on this) focuses on arbitrary \(p\)-value cutoffs for determining statistical significance and sometimes publication decisions. These problems are not the fault of \(p\)-values but, rather, the hyperfixation on the reject/retain decision for arbitrary test levels like \(\alpha = 0.05\). It might be best to view \(p\)-values as a transformation of the test statistic onto a common scale between 0 and 1. Finally, remember that \(p\)-values can only determine the statistical significance of an estimate, not its practical or substantive significance.

Warning

People use many statistical shibboleths to purportedly identify people who don’t understand statistics, and these criticisms sometimes hinge on seemingly subtle differences in interpretation that are easy to miss. If you have intuitively mastered the core concepts, however, avoiding these common pitfalls will be much easier.

The shibboleth with \(p\)-values is that sometimes people interpret them as “the probability that the null hypothesis is true.” But this doesn’t make sense from our definition because the \(p\)-value conditions on the null hypothesis—it cannot tell us anything about the probability of the null hypothesis being true. A more useful metaphor is that hypothesis tests are statistical thought experiments and that \(p\)-values answer the question: how likely would a result at least as extreme as ours be if the null were true?

4.5 Power analysis

Imagine you have spent a large amount of your research budget on a big experiment that tests a new and exciting theory, but the results come back, and… you fail to reject the null of no treatment effect. This can happen under two possible states of the world: (1) the null is true, and you correctly failed to reject it, or (2) the null is false but the test had insufficient power to detect the true effect (that is, to allow you to reject the null). Because this is unwanted uncertainty after the fact, it is common for researchers to conduct power analyses before collecting data. These analyses forecast the necessary sample size to ensure you can reject the null under a hypothesized effect size. These hypothesized effect sizes are vital to this exercise and often come from prior studies or substantive knowledge about the domain.

Generally, power analyses involve calculating the power function \(\pi(\theta) = \P((X_1, \ldots, X_n) \in R \mid \theta)\) for different values of \(\theta\). It might also involve sample size calculations for a particular alternative, \(\theta_1\), the hypothesized treatment effect. In that case, we try to find the sample size \(n\) to make the power \(\pi(\theta_1)\) at least as large as a particular value (often 0.8). For Wald tests, solving for the sample size is straightforward. For more general situations, however, we typically need numerical or simulation-based approaches to find the optimal sample size.

With Wald tests, we can characterize the asymptotic power function quite easily. Let \(\pi_n(\theta)\) be the power function for the Wald test at sample size \(n\).

Theorem 4.2 Suppose we have an asymptotically normal estimator that converges to an alternative \(\theta_1 \neq \theta_0\). That is, \(\sqrt{n}(\widehat{\theta}_n - \theta_1) \indist \N(0, V_{\theta_1})\) and \(\widehat{V}_\theta\inprob V_{\theta_1}\). A two-sided Wald test is consistent in the sense that \(\pi_n(\theta_1) \to 1\) as \(n \to \infty\). Furthermore, in large samples, its power is approximately \[ \pi_n(\theta_1) \approx 1 - \Phi\left( \frac{\theta_0 - \theta_1}{\sqrt{V_{\theta_1}/n}} + z_{1-\alpha/2} \right) + \Phi\left( \frac{\theta_0 - \theta_1}{\sqrt{V_{\theta_1}/n}}-z_{1-\alpha/2} \right). \tag{4.1}\]

From this, we can see that larger sample sizes imply higher power, holding the alternative and \(V_{\theta_1}\) fixed.

From this approximate power function, we can back out an approximate sample size needed to achieve a certain level of statistical power for a certain alternative. In standard notation, \(\beta\) is a test’s probability of a Type II error, so let \(1-\beta\) be the desired statistical power and \(n_{\beta}\) be the sample size required to achieve it. To see how to derive this sample size, first let \(a=(\theta_0 - \theta_1) / \sqrt{V_{\theta_1}/n_{\beta}}\). When \(\theta_1 < \theta_0\) (so \(a\) is positive), \(\Phi(a + z_{1-\alpha/2})\) will approach 1 as \(a\) gets larger in magnitude, so the power will be roughly equal to \(\Phi(a - z_{1-\alpha/2})\). If \(\theta_1 > \theta_0\) (so \(a\) is negative), then \(\Phi(a - z_{1-\alpha/2})\) will approach 0 as \(a\) gets larger in magnitude, and so the power will be roughly \(1 - \Phi(a + z_{1-\alpha/2}) = \Phi(-a - z_{1-\alpha/2})\). Combining these two facts, we can write the asymptotic power as approximately \(\Phi(|a| - z_{\alpha/2})\), with the approximation getting more accurate as \(|a|\) grows. This fact allows us to simplify and rearrange Equation 4.1 to obtain the required sample size: \[ \begin{aligned} 1 - \beta &\approx \Phi\left( \frac{|\theta_0 - \theta_1|}{\sqrt{V_{\theta_1}/ n_{\beta}}}-z_{1-\alpha/2} \right) \\ \Phi^{-1}(1-\beta) &\approx \frac{|\theta_0 - \theta_1|}{\sqrt{V_{\theta_1}/ n_{\beta}}}-z_{1-\alpha/2} \\ n_{\beta} &\approx \frac{\left(z_{1-\beta} + z_{1-\alpha/2}\right)^2V_{\theta_1}}{(\theta_0 - \theta_1)^2}, \end{aligned} \tag{4.2}\] where we used the identity \(\Phi^{-1}(1-\beta) = z_{1-\beta}\). The opposite case (where \(\theta_{1}\) is much larger than \(\theta_{0}\)) actually ends up with the exact same required sample size, as symmetry would hopefully imply.

Example 4.5 (Sizing for desired power) Let’s return to Example 4.4 and assume that we need to size our experiment. In most cases, we will have equal-sized arms, \(n_1 = n_0\), and for the purposes of sizing we will assume that the variance of the outcome is the same across treatments, \(\sigma^2_1 = \sigma^2_0 = \sigma^2\). Under these two simplifying assumptions, we can write the variance of the difference in means estimator as \[ \V[\widehat{\tau} \mid \mathbf{D}] = \frac{2\sigma^2}{n_1}, \] so \(V_{\theta_1} = 2\sigma^2\).

Now, suppose we will be conducting the usual Wald test of the null hypothesis that \(\tau = 0\) with \(\alpha = 0.05\). We want to determine the arm size, \(n_{1, \beta}\), needed to achieve 80% power (\(1 - \beta = 0.8\)) on an alternative that is 0.2 standard deviations of the population outcome, \(\tau_{1} = 0.2\sigma\). Plugging these values into Equation 4.2, we obtain \[ n_{1,\beta} \approx \frac{(z_{0.8} + z_{0.975})^2 \cdot 2\sigma^2}{0.2^2 \sigma^2} \approx 392.4, \] so we would need \(n_{1,\beta}=393\) per arm to achieve 80% power against this alternative.

4.6 Exact tests

An exact test is a statistical hypothesis test in which the distribution of the test statistic is fully known if the null hypothesis is true. Contrast this with the Wald test, which generally relies on asymptotic approximations to determine the distribution of the test statistic. We now turn to two different ways to construct exact tests: via randomization and via a parametric model like the normal.

4.6.1 Randomization tests

We opened the chapter with the example of the Lady Tasting Tea, which feels very distinct from the hypothesis tests of the last few sections. In fact, the Lady Tasting Tea example relies not on asymptotics but on the core idea of randomization.

Example 4.6 Let’s return to Fisher’s example of the Lady Tasting Tea and make the statistical test more explicit. Let \(T\) be the number of milk-first cups that Bristol labels correctly, which can take on values 0, 1, 2, 3, or 4. The null hypothesis is that Bristol cannot taste the difference between milk-first and tea-first. If this is true, her guesses would be the same no matter which cups had milk poured first. Thus, the number of correct guesses, \(T\), would be entirely determined by which 4 of the 8 cups Fisher and Roach selected to pour milk first. Crucially, they selected the cups to pour milk first randomly, so that all \(\binom{8}{4} = 70\) possible sets of milk- and tea-first cups were equally likely.

Table 4.2: Number of milk- vs tea-first cup arrangements matching the given number of Bristol’s guesses under the null hypothesis

T	0	1	2	3	4
Number of arrangements	1	16	36	16	1

Under the null, we can count how many of the 70 arrangements would match her labels. Table 4.2 shows these counts. If Bristol had labeled 3 cups correctly, the \(p\)-value would be \(\P(T \geq 3 \mid H_0) = 17 / 70 \approx 0.24\), which is fairly likely. In reality, Bristol did guess all four correctly, so our observed test statistic is \(T_{\textsf{obs}} = 4\). Under the null, there is only one out of the 70 possible cup arrangements that would produce a perfect set of guesses. Thus, the \(p\)-value in this case is \(\P(T \geq 4 \mid H_0) = 1 / 70 \approx 0.014\). Notice how the null distribution of the test statistic here arises from the randomization of the cups rather than a model of how Bristol was guessing.

Finally, notice how the discrete nature of the problem limits the \(p\)-values that are possible. From 3 to 4 correct guesses, the \(p\)-value jumps down from 0.24 to 0.014 and the latter is the smallest possible \(p\)-value we can obtain. To obtain smaller possible \(p\)-values and a finer scale of \(p\)-values, we could increase the number of cups in the test.

We can generalize the idea of the Lady Tasting Tea to a broad class of tests. A randomization test is a hypothesis test that leverages a randomization hypothesis that, under the null hypothesis, certain known transformations of the data have no effect on the distribution of the data. Then, under the null hypothesis, if we apply this transformation to the data, the distribution of the test statistic does not change. We can obtain a \(p\)-value that measures how likely it is to see a test statistic at least as extreme as the observed test statistic among all possible transformations.

Example 4.7 (Two-sample permutation test) Let’s return to the RCT example with pairs of outcome and treatment status variables \((Y_i, D_i)\). In Example 4.4 we conducted a hypothesis test of whether the mean of \(Y_i\) was different between \(D_i = 1\) and \(D_i = 0\) groups. A stronger null hypothesis is that the treatment and control observations come from the same distribution. Under this null hypothesis and randomization of \(D_i\), the statistic \(T\) of Example 4.4 has the same distribution even if we were to shuffle or permute the treatment statuses across units.

Enumerating all possible permutations of \((D_1, \ldots, D_n)\) is computationally infeasible even for moderate \(n\), so we often turn to the following simulation-based approach:

1. Calculate the observed test statistic, \(T_0:=T_{\textsf{obs}}\).
2. For a large number of repetitions, \(b = 1,\ldots, B\):
   1. Draw a random reshuffling of treatment status as \((D^*_1, \ldots, D^*_n)\).
   2. Calculate the test statistic under this permuted data, \(T_b\).
3. Compute the \(p\)-value as the fraction of all reshuffles (and the observed test statistic) that are at least as large in magnitude as the observed test statistic: \[ p = \frac{1}{B+1} \sum_{b=0}^B \mathbb{I}(|T_b| \geq |T_0|). \]
4. Reject the null if \(p \leq \alpha\).

Whatever \(B\) we choose, this test will have level \(\alpha\) since under the null, the observed test statistic is equally likely to land at any rank among the \(B+1\) values. The probability it is in the top \(k\) values is \(k / (B+1)\), so the probability that the \(p\)-value is less than or equal to \(k / (B+1)\) is exactly \(k/(B+1)\). Thus, rejecting when \(p \leq \alpha\) can never occur more than \(\alpha\) of the time under the null. Note that this argument assumes no ties between values of the test statistic, though ties only make the test more conservative not invalid.

Increasing \(B\) allows us to get closer to the full-enumeration \(p\)-value and gain a little power on average.

As these examples show, randomization tests are exact tests: under the null, the randomization completely determines the distribution of the test statistic, meaning we can guarantee the Type I error rate will be no larger than \(\alpha\) without any large-sample approximation. Thus, these tests rely on the design of the study rather than asymptotics to justify their properties, which is a distinct advantage. But notice that the comparison isn’t apples to apples. In Example 4.7, our null was that the entire distribution of \(Y_i\) was the same in the treated and control groups, which could be violated by, for example, differences in the variance of the two distributions. In Example 4.4, the large-sample approximation allowed us to focus a test on the null hypothesis of the difference in means alone. Both could be interesting hypotheses to test, but it is important to keep them distinct in our minds.

4.6.2 Exact tests under normal data

Another way to obtain an exact test is to assume a parametric model for the data. A popular model is the normal model where \(X_1,\ldots,X_n\) are iid samples from \(\N(\mu,\sigma^2)\). Under a null hypothesis of \(H_0: \mu = \mu_0\), we can show that \[ T_n = \frac{\Xbar_n - \mu_0}{S_n/\sqrt{n}} \sim t_{n-1}, \] where \(t_{n-1}\) is the Student’s t-distribution with \(n-1\) degrees of freedom. This result implies the null distribution is \(t\), so we use quantiles of \(t\) for critical values, at any sample size with \(n > 1\). For a one-sided test, \(c = G^{-1}_0(1 - \alpha)\), but now \(G_0\) is \(t\) with \(n-1\) df and so we use qt() instead of qnorm() to calculate these critical values.

The tail quantiles of the \(t\) distribution (and thus the critical values) are always larger in absolute value than the normal because the \(t\) distribution has fatter tails, as shown in Figure 4.3. The dashed lines show the 0.95 quantile that would be the critical value for a one-sided test with \(\alpha = 0.05\). We can see that the quantile for the \(t\)-distribution is larger than for the normal. As \(n\to\infty\), however, the \(t\) converges to the standard normal, and so it is asymptotically equivalent to the Wald test but more conservative in finite samples. Many software packages use this \(t\)-test as the default procedure because it allows for exact inference in the normal model. But given its asymptotic equivalence to the Wald test, we can (and many do) view it as an ad hoc conservative adjustment to the Wald test when the normal model does not hold.

Figure 4.3: Normal versus t distribution. Dashed lines are the 0.95 quantile of each distribution.

4.7 Multiple testing

The elegant theory and statistical guarantees of hypothesis testing rest on the presumption that such tests are done in isolation. If you conduct multiple hypothesis tests and accept any rejection of the null as a discovery, the level of the hypothesis test can be very misleading.

Example 4.8 If we design a series of 20 independent randomized controlled trials, we would conduct the usual hypothesis test in each trial for a total of 20 tests. If each of these tests has level \(\alpha\), then if we do each of these tests in isolation, each test will have, at most, an \(\alpha\) chance of a false positive under the null.

In Figure 4.4, we simulate this setup, drawing 20 null effects repeatedly and calculating the percentage of times we reject the null. In isolation, the rejection rates for each of the 20 tests are very close to the nominal level of 0.05. However, if we adopt a different rule in which we claim a discovery if any of the tests are significant, the rejection rate skyrockets to over 60%!

Figure 4.4: Rejection rates for twenty tests in isolation vs combining the tests to reject whenever any test is significant. Tests have level \(\alpha = 0.05\) which is shown as a dashed line.

4.7.1 Family-wise error rate

Formally, let \(H_{0,1}, \ldots, H_{0,m}\) be a series of null hypotheses we are testing with calculated \(p\)-values, \(p_1, \ldots, p_m\). We can describe the set of results across these tests in a \(2\times 2\) contingency table with margins.

Table 4.3: Outcomes when testing \(m\) null hypotheses

	\(H_0\) true	\(H_0\) false	Total
Reject \(H_0\)	\(W\)	\(U\)	\(K\)
Do not reject \(H_0\)	\(m_0 - W\)	\((m - m_0) - U\)	\(m - K\)
Total	\(m_0\)	\(m - m_0\)	\(m\)

In Table 4.3, \(m_0\) is the number of these null hypotheses that are true (which is a fixed parameter), \(W\) is the number of false discoveries (rejecting when the null is true), \(U\) is the number of true discoveries (rejecting when the null is false), and \(K = W+U\) is the total number of rejections/discoveries. The family-wise error rate (FWER) is the probability of making at least one false discovery, \(1 - \P(W=0)\).

There are various procedures to control the FWER, but the most popular is the Bonferroni correction. Recall that we can implement an \(\alpha\)-level test by rejecting the null hypothesis when \(p_i \leq \alpha\). The Bonferroni correction moves this rejection rule so that we reject any of the null hypotheses at \(p_i \leq \alpha / m\), which controls the FWER since \[\P(W\ge1)\le\sum_{i:\,H_{0,i}\text{ true}}\P(p_i\le\alpha/m)\le \frac{m_0\alpha}{m}\le\alpha.\]

For Example 4.8, we had \(m=20\) and \(\alpha = 0.05\), which means the Bonferroni-corrected cutoff would be \(0.0025\). Bonferroni is a very conservative correction since it is loose when there are null hypotheses that are false so that \(m_0 < m\) or when tests are positively correlated. This conservativeness means Bonferroni can have lower statistical power.

There are other, more powerful ways to control the FWER, including Holm’s procedure (Holm 1979). First, we order the \(p\)-values and null hypotheses in terms of their \(p\)-value rank, \(p_{(1)}, \ldots, p_{(m)}\) and \(H_{(1)}, \ldots, H_{(m)}\), so that \(p_{(i)}\) is the \(i\)th smallest \(p\)-value and \(H_{(i)}\) is its corresponding null hypothesis. Then we follow a step-down procedure:

• Reject \(H_{(1)}\) if \(p_{(1)} \leq \alpha / m\); otherwise, stop the procedure.
• Reject \(H_{(2)}\) if \(p_{(2)} \leq \alpha / (m - 1)\); otherwise, stop the procedure.
• Continue comparing \(p_{(k)}\) to \(\alpha / (m + 1 - k)\) and rejecting as above, stopping the procedure at the first \(p\)-value that goes above its threshold.

Clearly, the Holm procedure will reject any null hypothesis that Bonferroni will and potentially more, but it is possible to show that it also controls the FWER under any form of dependence across tests.

Two additional points are worth mentioning. First, Bonferroni and Holm both control the FWER in the strong sense—that is, \(\P(W \geq 1) \leq \alpha\) for any configuration of true and false null hypotheses. Second, both procedures assume that the \(p\)-values are valid in the sense that \(\P(p_i \leq u)\leq u\) under \(H_{0,i}\), so for a set of asymptotic tests, these results would be approximate.

4.7.2 False discovery rate

Controlling the FWER is fairly stringent because you are controlling any false discovery in a set of tests. This approach ignores what proportion of the discoveries are false. FWER treats two false discoveries out of five discoveries the same as two false discoveries out of 100 overall discoveries. This strictness can lead to low statistical power.

The false discovery rate is the expected fraction of discoveries (that is, rejections) that are false, or \(\text{FDR} = \E[W/K]\). To avoid division by zero, we set \(W/K = 0\) if there are no discoveries (\(K = 0\)). One popular method for controlling the FDR at level \(\alpha\) is the Benjamini–Hochberg (BH) procedure (Benjamini and Hochberg 1995), which works as follows:

1. Find the largest \(k\) such that \(p_{(k)} \leq \frac{k}{m} \alpha\).
2. Reject all null hypotheses, \(H_{(i)}\), with that or a smaller \(p\)-values, \(i = 1, \ldots, k\). If there is no such \(k\), reject none of the null hypotheses.

Under the assumption of independent tests or some forms of positive dependence, the BH procedure controls the FDR: \[ \text{FDR} \leq \frac{m_0}{m}\alpha \leq \alpha. \]

There are modifications to the BH procedure, such as Benjamini–Yekutieli (Benjamini and Yekutieli 2001), that can make it valid more generally. Another way to view the BH procedure is to plot the sorted \(p\)-values, \(p_{(i)}\), against their rank, \(i\), along with a line with slope \(\alpha/m\). Next, find the largest \(p\)-value on or below that line and reject it and all lower \(p\)-values. The next example demonstrates this approach.

Example 4.9 Returning to Example 4.8, suppose that some of the \(m=20\) trials actually have true effects. Furthermore, imagine Figure 4.5 shows the \(p\)-values we received in one instance of the 20 trials. For \(\alpha = 0.05\), the plot also shows which \(p\)-values would be rejected by the various procedures. The smallest is less than \(\alpha / m = 0.0025\) and would be rejected by all the methods: Bonferroni, Holm, BH, and the standard (uncorrected) testing procedure. Bonferroni and Holm reject nothing else since the next highest \(p\)-value at 0.0066 is above the Bonferroni cutoff of \(\alpha / 20 \approx 0.0025\) and the Holm cutoff of \(\alpha / 19 \approx 0.0026\).

The blue dashed line has slope \(\alpha / m = 0.0025\) and the fifth \(p\)-value is the largest one on or below this line, meaning the first five \(p\)-values would be rejected by Benjamini–Hochberg (even though the second is above the BH line). Finally, the sixth and seventh \(p\)-values would be rejected by standard testing but not by any of the corrections.

Figure 4.5: Rejection status for multiple tests with different multiple testing correction strategies. The solid line shows the Bonferroni cutoff of \(\alpha / m\); the dashed line shows the BH reference line \(k\alpha / m\); and the dotted horizontal line is the usual \(p\)-value cutoff of \(\alpha\).

4.8 Summary

In this chapter, we covered the basics of hypothesis tests, which are a type of statistical thought experiment. We assume a particular state of the world (namely, that the null hypothesis is true) and determine how unlikely a test statistic at least as extreme as the one we observed would be in that world. We described different types of tests (one-sided versus two-sided), introduced the properties of tests (size and power), and showed how to determine the rejection region of a test. We also described the Wald test, a general test that can be used in a wide variety of settings. \(p\)-values are a continuous measure of divergence between the observed data and the null hypothesis. Power analyses allow researchers to forecast how large a sample they will need to detect different effect sizes with sufficient statistical power. We discussed two approaches to hypothesis testing with exact level: randomization tests and the normal model. Finally, we saw how conducting several tests increases the probability of at least one false positive. We discussed the Bonferroni and Holm corrections that control family-wise error rate and the Benjamini–Hochberg procedure that controls the more lenient false discovery rate.

Next in Chapter 5, we will look at another way to think about uncertainty in our estimates: confidence intervals. As we will see, there is a deep connection between confidence intervals and hypothesis tests.

Benjamini, Yoav, and Yosef Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society: Series B (Methodological) 57 (1): 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.

Benjamini, Yoav, and Daniel Yekutieli. 2001. “The Control of the False Discovery Rate in Multiple Testing Under Dependency.” The Annals of Statistics 29 (4): 1165–88. https://doi.org/10.1214/aos/1013699998.

Box, Joan Fisher. 1978. R. A. Fisher: The Life of a Scientist. Wiley.

Fisher, Ronald A. 1935. The Design of Experiments. Oliver; Boyd.

Holm, Sture. 1979. “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics 6 (2): 65–70. http://www.jstor.org/stable/4615733.

Senn, Stephen. 2012. “Tea for Three: Of Infusions and Inferences and Milk in First.” Significance 9 (6): 30–33. https://doi.org/10.1111/j.1740-9713.2012.00620.x.

Wasserstein, Ronald L., and Nicole A. Lazar. 2016. “The ASA Statement on p-Values: Context, Process, and Purpose.” The American Statistician 70 (2): 129–33. https://doi.org/10.1080/00031305.2016.1154108.

---

1. The story here comes from Box (1978, 134), in which Bristol stated a preference for milk poured first, but sources vary in the details. Senn (2012) says Bristol preferred her cup with tea first. Fisher’s own presentation of the design (Fisher 1935, ch. 2) introduces the setting as a generic illustration without any of the historical details.

2. Different people and different textbooks describe what to do when we do not reject the null hypothesis differently. The terminology is not so important so long as you understand that rejecting the null does not mean the null is logically false and that “accepting” (or failing to reject) the null does not mean the null is logically true.