Multiple statistical tests: Lessons from a d20

Introduction

In scientific research, it is important to consider the issue of conducting multiple statistical tests and the likelihood of spuriously obtaining a ‘significant’ effect. Within a null-hypothesis significance testing (NHST) framework, statistical tests are usually conducted with α = .05, i.e., the likelihood of falsely rejecting the null hypothesis as .05. Interestingly, this value coincides with the probability of obtaining a specific outcome on a 20-sided dice (or ‘d20’), as $\frac{1}{20}$ = .05. In the current (fifth) edition of Dungeons & Dragons, a tabletop game, many in-game events are determined based on the outcome of a d20. However, to make some events more likely, there are times when players roll a d20 ‘with advantage’, meaning that they roll the d20 twice and take the greater value¹. (There are also instances where a d20 is rolled ‘with disadvantage’, where the lesser value is taken, but here I will only focus on the former case.) This parallels the use of NHST without any correction for multiple comparisons, as it is more likely to get a significant effect due to chance (i.e., Type-I error) if many tests are conducted without a correction for multiple comparisons.

Here I wondered how much the probability of obtaining a 20, on a d20, would increase due to multiple tests–i.e., obtaining at least one 20 across n die. This approach assumes that each statistical test is wholly independent from each other, and thus is likely to over-estimate the effect related to conducting multiple statistical tests using variations in how the measures are calculated or the use of different, but correlated, measures. Nonetheless, this exploration is based in probability theory and mathematical derivations, rather than computational simulations, and can serve as an comprehensible primer in understanding the relationship between repeated statistical tests and probability distributions.

Developing an intuition of statistics and probability distributions is of particular importance as most people, both laymen² and scientists^3,4, have misconceptions about NHST. This is further compounded by critics of NHST, which often over-emphasize the limitations of the approach, e.g., see 4–6. By providing a comprehensible example of how repeated statistical tests can inflate chance likelihoods, I hope that these demonstrations can improve researchers’ intuitions regarding NHST. This approach is not contrary to those suggested by the use of confidence intervals and Bayesian statistics—which have become increasingly adopted across the life sciences, from medicine to psychology^7,8—but rather to improve comprehension of the characteristics of NHST.

Mathematical derivations

The probability that of a specific outcome occurring on each on n die, each with d sides is:

The probability of obtaining a specific outcome across n rolls of a d-sided die are listed in Table 1.

To develop some intuition of the effect of multiple die rolls, several simple cases can be considered.

For d = 2, i.e., a coin, the probability of obtaining at a heads when flipping one coin (n = 1) is $\frac{1}{2}$. The probability of obtaining a heads twice (with two coins, n = 2) is ${\left(\frac{1}{2}\right)}^2$ or $\frac{1}{4}$. In contrast, the probability of obtaining at least one heads when flipping two coins is $\frac{3}{4}$, as there are four possible outcomes ({HH, HT, TH, TT}) and three of them satisfy the criteria of ‘at least one heads’ ({HH, HT, TH}) and only one outcome does not ({TT}). This can more clearly be considered as the complementary event, where the probability is 1 − $\frac{1}{4}$, which resolves to $\frac{3}{4}$.

For d = 6, i.e., a ‘conventional’ six-sided die, the probability of obtaining any specific outcome is $\frac{1}{6}$. When considering multiple dice, it is again important to differentiate the probability of obtaining ‘obtaining the same specific outcome multiple times’, e.g., the probability of obtaining two sixes with two dice is ${\left(\frac{1}{6}\right)}^2$ = $\frac{1}{36}$, from the case of ‘obtaining at least one specific outcome across multiple dice’. To determine the probability of obtaining a specific out-come on any of multiple dice, the complementary event should again be considered, i.e., the probability of not obtaining that outcome on any of the die. For n = 1, the probability of not obtaining a specific outcome is $\frac{5}{6}$. Following from this, the probability of obtaining that specific outcome is 1 − $\frac{5}{6}$ or $\frac{1}{6}$. When n = 2, the probability of not obtaining a six on either of the dice is ${\left(\frac{5}{6}\right)}^2$, which resolves to $\frac{25}{36}$. The complementary event of obtaining ‘at least one six’ is 1 − $\frac{25}{36}$ or $\frac{11}{36}$. Here we can see that with two dice, the probability of obtaining at least one six (or any other specific outcome) is nearly doubled, from $\frac{6}{36}$ (i.e., $\frac{1}{6}$ with a single die).

For d = 20, i.e., a 20-sided die, the probability of obtaining any specific outcome is $\frac{1}{20}$ or .05. If n = 2 dice are rolled, the probability of obtaining at least one 20 is $\frac{39}{400}$ or .0975. If n = 10 dice are rolled, the probability of obtaining at least one 20 is ≈ .4013. With n = 20 dice, this increases further to ≈ .6415.

We can also consider a more general problem, the probability of obtaining an outcome of o or greater, on at least one of n d-sided die:

For instance, when rolling a six-sided die, the probability of obtaining a five or higher is $\frac{2}{6}$ (equivalent to $\frac{12}{36}$). Following from the same approach of calculating the complementary event, the probability of obtaining not obtaining any two specific outcomes across multiple dice is 1 − ${\left(\frac{4}{6}\right)}^2$, which resolves to $\frac{20}{36}$. Figure 1 and Table 2 show the probability of obtaining at least o on a d = 20 die, across n dice.

Figure 1. Probability (P) of obtaining at least an outcome o once, across n d20 die.

Discussion

While it is widely understood that multiple comparisons need to be corrected for, many would underestimate the degree of inflation in Type-I error associated with additional, uncorrected statistical tests. Critically, statistical procedures have been developed to correct for multiple comparisons (e.g., Bonferroni, Tukey’s HSD), see 9 for a detailed review. Nonetheless, the mathematical derivations presented here clearly illustrate the influence of multiple statistical tests on the likelihood of obtaining a specific outcome due to chance alone. These derivations and examples should be useful in providing a concrete example of the problem associated with uncorrected multiple comparisons and may prove useful as a pedagogical tool.