Amendments from Version 1

F1000Research

2046-1402

F1000 Research Limited

London, UK

10.12688/f1000research.158434.2

Opinion Article

Articles

Addressing common inferential mistakes when failing to reject the null-hypothesis

[version 2; peer review: 2 approved]

Schmidt

Amand

Conceptualization Formal Analysis Methodology Visualization Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0003-1327-0424 a 1 2 3 4 1Department of Cardiology, University of Amsterdam, Amsterdam Zuidoost, 22660, Netherlands Antilles 2University College London Faculty of Population Health Sciences, London, England, UK 3Department of Cardiology, Utrecht University, Heidelberglaan, 3584 CX, Netherlands Antilles 4UCL British Heart Foundation Research Accelerator, London, Chenies Mews, WC1E6HX, UK

a amand.schmidt@ucl.ac.uk

Competing interests: AFS has received funding from New Amsterdam Pharma unrelated projects.

25 2 2025

2024

1488

7 2 2025

2025

This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Failure to reject a null-hypothesis may lead to erroneous conclusions regarding the absence of an association or inadequate statistical power. Because an estimate (and its variance) can never be exactly zero, traditional statistical tests cannot conclusively demonstrate the absence of an association. Instead, estimates of accuracy should be used to identify settings in which an association and its variability are sufficiently small to be clinically acceptable, directly providing information on safety and efficacy. Post-hoc power calculations should be avoided, as they offer no additional information beyond statistical tests and p-values. Furthermore, post-hoc power calculations can be misleading because of an inability to distinguish between results based on insufficient sample size and results that reflect clinically irrelevant differences. Most multiple testing procedures unrealistically assume that all positive results are false positives. However, in applied settings, results typically represent a mix of true and false positives. This implies that multiplicity corrections do not effectively differentiate between true and false positives. Instead, considering the distributions of p-values and the proportion of significant results can help to identify bodies of evidence unlikely to be driven by false-positive results. In conclusion, rather than attempting to categorize results as true or false, medical research should embrace established statistical methods that focus on estimation accuracy, replication, and consistency.

Statistical inference null-hypothesis equivalence testing statistical power accuracy.

AFS was supported by BHF

PG/22/10989

UCL BHF Research Accelerator

AA/18/6/34223

MR/V033867/1

National Institute for Health and Care Research University College London Hospitals Biomedical Research Centre.

AFS was supported by BHF grant PG/22/10989, the UCL BHF Research Accelerator AA/18/6/34223, MR/V033867/1, and the National Institute for Health and Care Research University College London Hospitals Biomedical Research Centre.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Revised Amendments from Version 1

Revised the manuscript to include additional details on the type of non-parametric test used and the sample size of the simulated data in Figure 5. Additionally, both the supplementary material and the manuscript were updated to provide proof of the relationship between p-values and observed power.

Background

Statistical tests and p-values are used to estimate the compatibility of the available data with a specific null-hypothesis. While statistical tests and p-values are strongly embraced by applied researchers, statistical science has raised important concerns about their interpretability, which may lead to incorrect statements about the presence, absence, or importance of an association. ¹ This has led some researchers to suggest the replacement of p-values with alternative metrics, such as the S-value (i.e., S for surprise). ^{2,
3} The current manuscript attempts to provide an accessible overview of these voiced concerns, particularly focusing on the appropriate interpretation of “non-significant” results when a p-value or statistical test does not provide sufficient reason to reject the posed null-hypothesis. In line with previous guidance in this area, this manuscript suggests moving away from categorizing results as True or False, instead suggesting research focusses on obtaining sufficiently accurate results on potential benefits and harms.

As an illustrative example, we will consider results from the VOYAGER PAD ⁴ trial, which randomized patients with peripheral artery disease (PAD) to twice-daily rivaroxaban (2.5 mg) or a placebo, evaluating differences in the incidence of ischemic cardiovascular disease. The reported hazard ratio (HR) for rivaroxaban was 0.85 (95%CI 0.76; 0.96), tested against a null-hypothesis HR of 1.00, which resulted in a p-value of 0.006. Here the p-value indicates the proportion of subsequent trials (using the same design, intervention, and types of patients) which would result in an HR of 0.85 or more extreme, assuming the true population HR is 1. By convention, a p-value smaller than 0.05 is considered “significant”, however more stringent or liberal choices may also be applied. Aside from whether the CI rejects an HR of 1, the size of the 95%CI provides an indication of the variability of the HR, which in this case supports a smaller (HR of 0.96) or larger (HR of 0.76) benefit. Irrespective of the effect magnitude, there is reason to question the null-hypothesis HR of 1, where the data provides most support for an HR of 0.85.

In the aforementioned example, there is substantial evidence against the null-hypothesis, and hence, inference is fairly straightforward and uncontested. Conversely, interpreting results of “non-significant” analyses, where the null-hypothesis cannot be rejected, may leads to erroneous conclusions such as claiming a lack of “statistical significance” supports the null-hypothesis. For example, based on an HR of 0.86 (95%CI 0.40;1.87; p-value=0.71), the VOYAGER-PAD authors concluded that for the subgroup of patients with endovascular PAD, there was “no increase in intracranial or fatal bleeding”. ⁵ While it is clear that the null-hypothesis of no difference cannot be rejected, with the CI including an HR of 1.87, it is also clear that there is little evidence to support the absence of a harmful effect. Instead of claiming an absence of a risk-increasing effect, the presented results suggest that additional research is needed before drawing conclusions on bleeding risks.

In the following, we will discuss three common mistakes when interpreting results from statistical tests that fail to reject the null-hypothesis: 1) claiming that the null-hypothesis is true, 2) claiming that the study was underpowered, and 3) using multiple testing corrections to support claims about true or false associations.

Why a non-significant result does not rule out a potentially meaningful association

There are two types of hypotheses: a strict null-hypothesis, where a supposed population parameter μ (e.g., the mean, mean difference, or hazard ratio) takes on a single value (e.g., μ 0 = 0 ), and a composite null-hypothesis, where μ follows a range of values (e.g., 0.9 < μ 0 < 1.1 ).

As with the illustrative examples, most often a strict null-hypothesis is (implicitly) evaluated. Given that a strict null-hypothesis postulates that μ is equal to a single value, it can be readily demonstrated that a statistical test cannot support the strict null-hypothesis. This understanding stems directly from the fact that for an arbitrary function ∫ its integral sum with limits a = b is zero (see Extended data for a formal proof ). More intuitively, to prove that μ is equal to any single value, one must obtain an estimate with zero bias and infinite precision. Essentially, this requires divine knowledge about the value of μ .

While a strict null-hypothesis cannot be supported by the data, no matter how much data is collected, there may often be a need to rule out certain effect(s). For example, the risk of bleeding when treating patients with rivaroxaban, or identifying interventions with limited efficacy. The solution is to simply use composite null-hypotheses. Returning to the illustrative example, one might consider a risk-increasing effect of 1.25 or less as sufficiently modest to not offset the observed benefit on ischemic cardiovascular disease. In this case, the null-hypothesis would be μ 0 ≥ 1.25 , which would be rejected when both the estimated HR and the upper bound of the CI are smaller than 1.25 (or similarly using a one-sided test). For example, estimate 2 in Figure 1 reflects the VOYAGER PAD estimate of the rivaroxaban bleeding effect, and it is clear that the confidence interval includes HRs above 1.25; hence, the null-hypothesis should not be rejected and there is no reason to conclude that rivaroxaban has a relatively small effect on bleeding risk. However, estimate 4 is HR 1.15 (95%CI 1.06; 1.24) does exclude a HR larger than 1.25. Thus while estimate 4 supports a risk increasing effect, one can nevertheless conclude that the effect size is sufficiently small to suggest that rivaroxaban is relatively safe.

Figure 1. A forest plot illustrating non-inferiority and equivalence testing.

N.b. Points represent hazard ratios (HR) with horizontal lines indicating 95% confidence intervals (CI). The orange vertical line indicates margins of equivalence at 0.80 and 1.25, with the vertical line at 1.00, indicating no difference. Estimates 1-2 are based on the VOYAGER PAD ⁵ rivaroxaban results for ischemic cardiovascular disease and bleeding risk, respectively. Estimates 3-5 are purely hypothetical and included as illustrations.

This procedure is referred to as non-inferiority testing, where 1.25 is the bound of equivalence. Depending on what is deemed clinically non-inferior, such a bound can be substantially larger or smaller. For example, in the EBBINGHAUS trial of the PCSK9-inhibitor evolocumab the equivalence bound was set equal to 20% of the standard deviation of the cognitive function score measured in the placebo group. ⁶ The key characteristic of non-inferiority testing for safety is that both the point estimate (here, the HR) and the upper bound of the CI should be smaller than the supposed bound of equivalence. Given that one is not testing against a strict null-hypothesis of no difference, whether confidence (or statistical test) refutes a neutral value (e.g. an HR of 1) is immaterial. The consideration of two bounds, one on each side of a neutral HR of 1, naturally leads to equivalence testing. For example, in Figure 1, both estimates 4 and 5 are deemed equivalent, despite both rejecting an HR of 1.0.

Defining bounds of equivalence or non-inferiority is challenging and a possible source of contention. Typically, such bounds are defined by combining statistical and clinical considerations. For example, evidence from previous studies can be meta-analysed to obtain a pooled effect estimate and confidence interval, where the confidence interval limits can be multiplied by a constants representing the amount of effect that one would like to preserve or rule out (for safety). ⁷

Ideas about non-inferiority and equivalence can be further generalized by considering the entire range of CIs providing information on precision and the HRs supported by collected data ² ( Figure 2). For example, the rivaroxaban trial data support a wide range of bleeding risk HRs, and an HR of 1.25 would only be excluded using a 66%CI ( Figure 2, left panel). This can be contrasted by the small range of HRs supported by the rivaroxaban trial data for a protective effect on ischemic cardiovascular disease, indicating that the trial data are highly supportive of a protective effect ( Figure 2, right panel).

Figure 2. A compatibility graph comparing the confidence interval coverage against a range of hazard ratios for the rivaroxaban estimates on ischemic cardiovascular disease (left) and bleeding outcomes (right).

Vertical lines indicate a hazard ratio (HR) of 1.00, and a possible margin of equivalence for an HR of 1.25. The shaded area indicates the HRs supported for a given coverage probability, indicated on the y-axis.

Why post-hoc power provides the same information as a p-values and null-hypothesis test

Failure to reject a null-hypothesis naturally raises concerns about whether the study was sufficiently powered to detect a difference, often tempting researchers to conduct “post-hoc” or “observed” power calculations utilizing the observed point estimates (e.g. HRs) and variance estimates. Briefly, power reflects the probability of rejecting the null-hypothesis if it is false. This is the direct opposite of statistical tests and related quantities, such as p-values, which reflect the rejection probability assuming the null-hypothesis is true.

As such, p-values and observed power are equivalent, and no additional information is obtained by considering both ( Figure 3, see Extended data for a formal proof). To see this, suppose we observe a p-value of 0.05, which is equal to the conventional level of significance indicated by an alpha (i.e., the type 1 error rate) of 0.05. In this case, observed power can be obtained by calculating as probability of rejecting the null-hypothesis assuming the point estimates and it standard error are the true population values (which implies that the null-hypothesis is false), which in this case would be exactly 50% ( Figure 3). As such, while p-values evaluate the difference between the estimated values and the null-hypothesis assuming the former is true, power evaluates the same estimated values assuming the null-hypothesis is false. Hence, when p-values have been calculated, calculating observed power offers no additional information about the absence or presence of a difference.

Figure 3. The relationship between p-values and observed power.

Alpha refers to the type 1 error rate of a test. The dashed lines on the x-axis indicate the locations where the p-value is exactly equal to the alpha. The dashed lines on the y-axis indicate an observed power of 50%.

More concerning is that post-hoc power calculations may lead to erroneous conclusions regarding the lack of statistical power or the absence of an effect. For example, the following two HR estimates both have a p-value of 0.71 and a power of 7%: HR 0.86 (95% CI 0.39; 1.91), HR 1.00 (95%CI 0.99; 1.01). Looking at the post-hoc power, one might conclude that both analyses were underpowered; however, the HR of 0.86 is non-significant due to considerable variability, whereas the latter HR of 1.00 simply reflects a clinically irrelevant association. Hence, a more relevant alternative to post-hoc power calculations is to evaluate the extent to which the CI includes clinically relevant effect estimates, which is in line with the aforementioned equivalence/non-inferiority approach. For example, the VOYAGER PAD HR estimate of 0.86 (95%CI 0.40;1.87) for bleeding risk in people with endovascular PAD clearly shows that the collected data supports a wide range of effect estimates, including potentially harmful associations. However, because the confidence interval only partially overlap with the proposed (hypothetical) upper bounds of acceptable harm of 1.25, testing against this bound results a p-value of 0.17 which is considerably smaller than testing against the complete absence of an effect: p-value 0.70. By comparison, the observed power estimate for these results is 7%, which implies that if the true HR was 0.86 one would have rejected the strict null-hypothesis in 7 out of 100 repeated experiment. As such observed power provide limit information relative to the presented alternative approaches, particularly the confidence interval based approach which allows for an informative discussion of benefits and harms in terms of effect magnitude(s).

Researchers may alternatively wish to calculate the power to reject a clinically meaningful difference other than the point estimate. Such calculations can meaningfully inform the design and viability of future studies; although sample size estimates may be more readily interpretable. However, when such power calculations are used to make statements about the presence or absence of an effect, or even lack of sample size, the described approach utilising confidence intervals and equivalence/non-inferiority margins provides more relevant information on study accuracy.

The futility of using power to make claims of the absence of an effect is further illustrated by noting that in the absence of an effect (i.e. when the p-value is 1) observed power is equal to the employed alpha threshold (e.g. 0.05). Hence, rather counterintuitively, low power may actually argue for the absence of an effect. Because power can only be calculated assuming the null-hypothesis is false, this metric cannot be used to make claims in favour of the null-hypothesis. Furthermore, as discussed in the preceding section, statistical tests cannot be used to support the strict null-hypothesis, as such this also holds for derived metrics such as p-value and power. While power remains essential when designing a future study, it should not be used to interpret results of a completed study. At this stage more relevant metrics such as confidence intervals are available which do not condition on the presence or absence of an effect, and provide information on accuracy as well as on effect magnitude.

Why multiple testing correction does not differentiate between true and false results

Considerations of power and type 1 error rate are extremely relevant when designing a study, ensuring that a sufficiently accurate effect estimate may be realistically obtained given the available resources. However, both power and type 1 error rate are conditional probabilities assuming that the null-hypothesis is either true or false. As such these concepts are less relevant after the data have been collected, which would generally not consist of null-hypothesises which are either all true or all false, but instead will include an unknown mixture of both. It is important to realize that the type 1 error rate itself does not reflect the proportion of false positive results, but merely reflects the expected proportion of false positive results should all null-hypotheses be true.

When considering the results of two or more null-hypothesis tests, in an attempt to decrease the number of false positive results, there is an expectation to perform multiple testing corrections. For example, the Bonferroni method is a popular multiplicity correction evaluating p-values against an alpha (e.g., 0.05) divided by the number of conducted tests. It is well known that post-hoc multiple testing correction (e.g., corrections not accounted for during the design stage by increasing the collected sample size) will decrease power ( Figure 3). An often overlooked point is that, depending on the unknown balance between false positives and true positives in a set of test results, applying multiplicity correction can sometimes increase the false discovery rate (i.e., the fraction of false positives divided by the total number of positive tests) instead of reducing it. For example, Figure 4A presents a naive expectation of multiple testing corrections, where the false discovery rate decreases from 1/3 to 0. However, there is no reason why the scenario depicted in Figure 4B may not occur; here, the false discovery rate increases from 1/3 to 1.

Figure 4. Illustrating the impact of multiple testing correction on the false discovery rate.

In each panel, four tests are true positives (orangered) and six are false positives (black). Horizontal lines are drawn at 0.05 and 0.005, the latter reflecting a Bonferroni correction for the 10 applied null-hypothesis tests.

As shown in Figure 5, in which the distribution of p-values is generated in the absence and presence of an association, small p-values may occur in both settings. As such, while a small p-value (or equivalently, a extreme test statistic) is unlikely to occur when there is no association, observing a single small p-value is insufficient to differentiate between true and false positive results. Concepts such as indirect or direct replication and internal consistency are more relevant to differentiate between true and false positives. For example, in the case of rivaroxaban, associations with multiple types of ischemic cardiovascular events (e.g., myocardial infarction, ischemic stroke, and acute limb ischemia) will be more convincing than an association with any one outcome.

Figure 5. The distribution of p-values when the null-hypothesis is true or false.

N.b. The p-values were derived by arbitrarily sampling 1,000 test statistic from a normal distribution and leveraging its cumulative density function to calculate the area on the left and right side of the sampled test-statistic. Specifically, the employed standard distribution had a standard deviation of 1 and mean of either 0 or 2, when the null-hypothesis was true and false, respectively. Please note that the normal distribution is only used as an exemplar, and alternative distributions with a known cumulative density function (e.g. chi-square, beta, or gamma) could have been used instead.

While individual p-values and null-hypothesis tests cannot differentiate between false and true positive results, a set of p-values ( Figure 5) can be compared against a uniform distribution to determine the likelihood that the entire set is driven by false positive results. This approach is independent of the specific statistical test used to derive individual p-values. Moreover, the method can be generalised to account for dependencies among p-values, such as dependencies arising from the inclusion of both composite and individual outcomes (e.g., evaluating both any stroke and ischaemic stroke). Similarly, one can determine the proportion of p-values that are smaller than a predefined alpha; for example, in Figure 5, the proportion of p-values smaller than 0.05 is of course 0.05 for the top panel and 0.49 for the lower panel. Returning to our illustrative example, despite showing a potentially protective association with myocardial infarction, ischemic stroke, major amputation, and venous thromboembolism, the null-hypothesis could only be rejected for the association between rivaroxaban and acute limb ischemia: HR 0.67 (95% 0.55; 0.82). ⁴ Utilizing a non-parametric Kolmogorov-Smirnov test to compare the set of p-values for all the aforementioned outcomes against a uniform distribution nevertheless resulted in a p-value of 0.02, suggesting that the protective effect of rivaroxaban is shared across multiple cardiovascular outcomes.

Discussion

In the current manuscript, we have addressed why statistical tests cannot be used to support the strict null-hypothesis. Instead, concerns regarding the safety or lack of efficacy should be evaluated using equivalence testing. This can be readily implemented in any study by combining confidence intervals with bounds of clinical insignificance. Such an approach provides direct information on whether a non-significant test is due to an association and its variability being sufficiently small or simply reflects a lack of accuracy. Furthermore, contrary to expectation and depending on the unknown proportion of true positive results, multiple testing corrections may increase the false discovery rate. Finally, because power and type 1 error rate make extreme assumptions about whether all results are true or false positives, these concepts have limited relevance after the data have been collected and analysed.

The current manuscript provides guidance on how standard null-hypothesis testing can be used to provide clinically meaningful insights, and attempts to move beyond the current erroneous modus vivendi, categorizing associations as true and false. Contrary to recent calls to completely abandon significance testing, ² this contribution calls for a more considerate and bespoke application of the currently available and ubiquitously accepted methods. Specifically, researchers should routinely indicate bounds between which an effect is sufficiently small to be considered clinically irrelevant. Related to this, the idea that any intervention should (or even can) be without harmful side effects needs to be dismissed and replaced with a notion of benefit versus harm, where clinically supported bounds off irrelevance can help to directly inform. Second, while notions of power and type 1 errors are essential at the study design phase, because these deal in hypothetical scenarios where all results are either true or false such metrics have limited relevance when interpreting results. Power and type 1 errors can be framed in terms of probabilities because the analysis has not yet been conducted. Once the experiment has been completed, these hypothetical probabilities are immaterial, and one is simply confronted with an unknown proportion of true-positive results. At this stage, concepts of power and type 1 error must be replaced by indicators of precision, such as confidence intervals. Instead of using confidence intervals as a proxy for null-hypothesis testing (i.e., whether the null-hypothesis value is excluded), inference should focus on determining too what extent there is sufficient precision to exclude meaningful differences. Finally, while decreasing the significance threshold (e.g. from 0.05 to 0.005) decreases the type 1 error rate this decreases power as well, and hence may decrease the number of true associations discovered. Depending on the area of research overlooked, true positive results may be more harmful than false positive results. For example, protein drug targets identified in early drug development are often subjected to a substantial number of follow-up analyses, which filter out false positive results. Such follow-up studies, however, rarely expand the number of candidates, hence suggesting a more inclusive approach might be more considerate. In settings more proximal to clinical implementation and less discovery oriented, such as phase 3 clinical trials, stringent multiple testing correction is clearly called for. Notwithstanding, it is important to realize that not every study needs to be designed as a clinical trial.

In conclusion, failure to reject a strict null-hypothesis does not support the absence of a clinically meaningful association. Instead, researchers should routinely apply composite null-hypothesis tests evaluated against meaningful bounds of insignificance. Genuine consideration of estimation accuracy, as provided through confidence intervals, precludes the need for questionable post-hoc power calculations. Finally, because power and type 1 errors unrealistically assume that all results are true or false, these concepts have limited value once data collection and analysis have been completed.

Author contributions

AFS designed the illustrations and wrote the manuscript.

Ethics and consent

Ethics and consent were not required.

Data availability

No data associated with this article.

Extended data

Addressing common inferential mistakes when failing to reject the null-hypothesis https://doi.org/10.5522/04/27854043. ⁸

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

References 1

Wasserstein

Lazar

: The ASA Statement on p-Values: Context, Process, and Purpose. Am. Stat. 2016;70:129–133. 10.1080/00031305.2016.1154108

Rafi

Greenland

: Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise. BMC Med. Res. Methodol. 2020;20:244. 32998683

10.1186/s12874-020-01105-9

PMC7528258

Cole

Edwards

Greenland

: Surprise!. Am. J. Epidemiol. 2021;190:191–193. 32648906

10.1093/aje/kwaa136

PMC7850156

Bonaca

: Rivaroxaban in Peripheral Artery Disease after Revascularization. N. Engl. J. Med. 2020;382:1994–2004. 10.1056/NEJMoa2000052

Rymer

: Rivaroxaban Plus Aspirin Versus Aspirin Alone After Endovascular Revascularization for Symptomatic PAD: Insights From VOYAGER PAD. Circulation. 2023;148:1919–1928. 37850397

10.1161/CIRCULATIONAHA.122.063806

Calabro

Gragnano

Pirro

: Cognitive function in a randomized trial of evolocumab. N. Engl. J. Med. 2017;377:1996–1997. 29143516

10.1056/NEJMc1712102

Althunian

Boer

de Groenwold

RHH

: Defining the noninferiority margin and analysing noninferiority: An overview. Br. J. Clin. Pharmacol. 2017 Aug;83(8):1636–1642. 28252213

10.1111/bcp.13280

PMC5510081

Addressing common inferential mistakes when failing to reject the null-hypothesis.

10.5522/04/27854043

10.5256/f1000research.177945.r368387

Reviewer response for version 2

Heimel

J. Alexander

1 Referee https://orcid.org/0000-0002-5291-4184 1Netherlands Institute for Neuroscience, Amsterdam, The Netherlands

Competing interests: No competing interests were disclosed.

21 3 2025

2025

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

recommendation

approve

The author has answered my comments to my satisfaction, except for one small point.

In his reply to my question, the author wrote that the relationship between p-value and observed power is independent of sample size n. I do not believe this to be true. In the new supplementary material, the author sketches this relationship, but ignores the fact the t-distribution (denoted as g in the supplementary material) is dependent on the degrees of freedom (d.o.f. = n-1) and therefore on the sample size. The value t_c will thus depend on n and the observed power, given by the author as g(t_0-t_c)+g(-t_0-t_c) will also depend on n, through g and t_c. In fact, I believe the correct formula for observed power should be:

Observed_power(t) = g_{d.o.f, t}(t_c) + g_{d.o.f, -t}(t_c)

With g_{d.o.f.,t}(t_c) the cumulative non-central t-distribution with d.o.f. degrees of freedom shift t evaluated at t_c.

For large n, the approximations used by the author converge, and the dependence on the sample size disappears.

This dependence on sample size, however, does not fundamentally change or invalidate the author’s argument that a post-hoc power calculation using the observed effect is not useful.

While studying the relationship between observed power and p-value for my reply, I came across the Wikipedia page on power analysis that also makes the author’s point that post-hoc power analysis is fundamentally flawed, with references to Hoenig (The American Statistician, 2001), cited more than 2000 times, and Thomas (Conservation Biology, 1997) already pointing out the same flaw. The author’s case is thus not new, but may still be very relevant and a good reminder. It certainly was for me.

Minor details about the new supplementary note:

I think that when writing “the p-value for a one-sided tests would simply be 1-g(|t_0|)”, already the assumption of f being symmetrical about zero is made. The statement is not true for a general f and t_0<0.

“In this case the two-sided p-value is 2x(1-g(2)) \approx 0.46”. I believe this should be “\approx 0.046” (and is only true for large sample size).

Is the topic of the opinion article discussed accurately in the context of the current literature?

Partly

Are arguments sufficiently supported by evidence from the published literature?

Partly

Are all factual statements correct and adequately supported by citations?

Partly

Are the conclusions drawn balanced and justified on the basis of the presented arguments?

Partly

Reviewer Expertise:

Neuroscience

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

References 1

: The Abuse of Power. The American Statistician .2001;55(1) : 10.1198/000313001300339897 19-24

10.1198/000313001300339897

Thomas, Len. "Retrospective power analysis." Conservation Biology 11.1 (1997): 276-280.

Schmidt

Amand

University College London Faculty of Population Health Sciences, London, England, UK

Competing interests: No competing interests were disclosed.

25 3 2025

Reviewer 2

Comment: The author has answered my comments to my satisfaction, except for one small point.

Observed_power(t) = g_{d.o.f, t}(t_c) + g_{d.o.f, -t}(t_c)

With g_{d.o.f.,t}(t_c) the cumulative non-central t-distribution with d.o.f. degrees of freedom shift t evaluated at t_c.

For large n, the approximations used by the author converge, and the dependence on the sample size disappears.

This dependence on sample size, however, does not fundamentally change or invalidate the author’s argument that a post-hoc power calculation using the observed effect is not useful.

Response: I am pleased to note that the reviewer agrees with the main conclusions of the manuscript.

The reviewer appears to be referring to the sample size–dependent convergence of the t-distribution to the standard normal distribution. While this is, of course, entirely correct and uncontested, it is not directly relevant to the point under discussion. The relationship between p-values and statistical power is independent of the choice of distribution, and does not rely on any asymptotic approximation.

Although the illustrative examples in the manuscript use the standard normal distribution -which does not depend on sample size or degrees of freedom - the relationship between p-value and power holds equally when using distributions that do, such as the t- or F-distributions. This is because both the p-value and power are calculated using the same distribution, with the same degrees of freedom. For example, if one calculates a p-value using a t-distribution with 7 degrees of freedom, the corresponding power would also be calculated using that same t-distribution with 7 degrees of freedom. Thus, while the distribution itself may depend on sample size, the relationship between p-value and power remains exact and independent of sample size.

The following was added to the supplementary to help clarify this.

“

As in the previous section, these expressions do not depend on the amount of data collected and are therefore exact. If one is uses a statistical distribution with degrees of freedom (which often depend on sample size), such as the t or F distributions, the relationship between the p-value and power remain exact, simply because both quantities are calculated using the same distribution with the same degrees of freedom.

“

Comment: While studying the relationship between observed power and p-value for my reply, I came across the Wikipedia page on power analysis that also makes the author’s point that post-hoc power analysis is fundamentally flawed, with references to Hoenig (The American Statistician, 2001), cited more than 2000 times, and Thomas (Conservation Biology, 1997) already pointing out the same flaw. The author’s case is thus not new, but may still be very relevant and a good reminder. It certainly was for me.

Response: Thank you these reference have been included in the updated manuscript.

Comment: Minor details about the new supplementary note:

Response: This section briefly describes p-values for one-sided and two-sided tests.

The section has been clarified to explain that this statement is not an assumption but merely reflects a known quantity about some distributions being symmetrical about zero where others are not. The expanded section now provides examples of alternative distributions which are not symmetrical about zero. The reason that this was not originally included is that commonly known non-symmetrical distribution such as the Chi-square, the F distribution, or even the Gamma distribution are not defined for negative values making them less applicable for two-sided tests.

“

Next, say we observe test-statistic t o , in that case the p-value for a one-sided tests would simply be 1-g t o , and assuming f is symmetrical about zero (e.g. the normal distribution or t-distribution) the p-value for a two-sided test would simply be 2× 1-g t o . For distributions that are not symmetric about zero, both sides of the distribution should be considered separately. It is important to note, however, that many commonly used asymmetric distributions—such as the F, Chi-squared, and Gamma distributions—are defined only for positive values. This restricts their direct applicability for two-sided hypothesis testing.

“

Comment: “In this case the two-sided p-value is 2x(1-g(2)) \approx 0.46”. I believe this should be “\approx 0.046” (and is only true for large sample size).

Response: Thanks for spotting this type-o, this should be 0.046 indeed.

However, this is not dependent on any sample size assumption. The approximate symbol (≈) is used purely to reflect rounding to three decimal places, not to imply asymptotic behaviour or large-sample properties.

It’s possible the reviewer is conflating this with the well-known approximation of the t-distribution by the standard normal distribution as sample size increases. While that is certainly valid in a different context, it is not relevant here. I am not making any claims about inference or distributional convergence; rather, I am simply referring to the numerical value of the cumulative distribution function.

To clarify: if g denotes the cumulative distribution function of the standard normal distribution, then 2×(1-g 2 ≈0.046, regardless of sample size. Similarly, if one were to use the t-distribution with 7 degrees of freedom, the corresponding value would be approximately 0.086 - the choice of 7 here being arbitrary and illustrative.

This is a purely mathematical statement about evaluating a distribution function at a specific point, and not a comment on sampling or estimation.

10.5256/f1000research.177945.r368388

Reviewer response for version 2

Cui

Ying

1 Referee https://orcid.org/0000-0002-3697-5155 1Stanford University, Stanford,, California, USA

Competing interests: No competing interests were disclosed.

6 3 2025

2025

recommendation

approve

No further comments.

Is the topic of the opinion article discussed accurately in the context of the current literature?

Yes

Are arguments sufficiently supported by evidence from the published literature?

Yes

Are all factual statements correct and adequately supported by citations?

Yes

Are the conclusions drawn balanced and justified on the basis of the presented arguments?

Yes

Reviewer Expertise:

Hypothesis testing, Biostatistics

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

10.5256/f1000research.174018.r350494

Reviewer response for version 1

Heimel

J. Alexander

1 Referee https://orcid.org/0000-0002-5291-4184 1Netherlands Institute for Neuroscience, Amsterdam, The Netherlands

Competing interests: No competing interests were disclosed.

7 1 2025

2025

recommendation

approve-with-reservations

The manuscript, “Addressing common inferential mistakes when failing to reject the null-hypothesis” contains two main messages. First, the author argues that post-hoc power calculations should be avoided, as “they offer no additional information beyond statistical tests and p-values” and they “can be misleading because of an inability to distinguish between results based on insufficient sample size and results that reflect clinically irrelevant differences”. Second, for interpreting the results of multiple comparison, the authors recommends to consider “the distribution of p-values and the proportion of significant results to identify bodies” rather than standard multiple testing procedures, because they “unrealistically assume that all positive results are false positive”. The paper is an interesting read, but I have a few remarks.

I have two comments on the first point.

“p-values and observed power are equivalent, and no additional information is obtained by considering both ( Figure 3).” No proof is given of this statement and it is not completely trivial. Can a (hint of a) proof be given or a reference with a proof cited? I believe the relationship shown and figure 3 between p and power depends on the number of samples, but this is not shown in the text or figure. Can you comment on this? The dependence on the sample number could give some added value for making this post-hoc power calculation, and p-values and observed power are thus not completely equivalent.

Are there examples that people do this particular post-hoc power calculation using the measured effect and the measured standard deviation? What I have encountered, is post-hoc power calculations based on the measured standard deviation of the sample, and for an assumed effect size (rather than the measured effect). If there are publications doing the post-hoc power analysis in the way suggested by this publication, then please give some example references. Otherwise, the reader is left to wonder if this part of the manuscript is arguing against hypothetical reasoning that people do not actually use.

The second point of the paper is that “While individual p-values and null hypothesis tests cannot differentiate between false and true positive results, a set of p-values (Figure 5) can be compared against a uniform distribution to determine the likelihood that the entire set is driven by false positive results.” This can indeed be helpful. However, if assumptions like independence between tests or normality of the measurements for some tests fail, then this meta-analysis on the p-value may give a false positive outcome. Without a good understanding of the underlying data and statistics, I would use this test only to suggest that results are possibly false positives if the distribution of p-values is not distinguishable from a uniform distribution, rather than concluding that there must be some true positive effect if the distribution of p-values is not uniform, as is done by the author for the specific example for the effect of rivaroxaban.

These are my main comments. I also have a number of smaller remarks.

The manuscript also discusses whether null hypothesis testing can prove whether difference in means between two (infinite) populations is truly zero, but I fail to see the connection of this more philosophical point on the interpretation of hypothesis testing to the main messages. I think that the manuscript would be stronger if it is left out.

The meaning of the sentence “Given that power and type 1 error make extreme assumptions where either all results are true or false positives, these concepts are less relevant after the data have been collected. ” is not at all clear to me. Perhaps rephrase to clarify.

“applying multiplicity correction can sometimes increase the false positive rate instead of reducing it.” In this sentence, and the ensuing section the term “false positive rate” is used differently from what is the common use in the statistical literature. In the literature, “false positive rate” is the expectation of a false positive result when repeating the procedure. The meaning that is taken in this manuscript is that of the fraction of the significant results of a given set of experiments that is a false positive. It is not clear that the interpretation chosen here, and which is shown by example to lead to a false expectation, actually often occurs in the scientific literature. Can some example references be provided?

Fig. 5: The number of samples used for producing the figure is missing.

“comparing the set of p-values for all the aforementioned outcomes against a uniform distribution resulted in a p-value of 0.02,” Which test is used for the comparison.

“researchers should routinely indicate the bounds between which an effect is sufficiently small to be considered clinically irrelevant” This is a valuable recommendation. Before the start of a study it is necessary to choose an expected effect size for determining the power of a study or the number of samples. If the expected effect size, is below the bound, then either the study should not be started (if one is looking to prove a positive effect) or the bound should be taken as the effect size for the power calculation.

“Second, while notions of power and type 1 errors are essential at the study design phase because these deal in hypothetical scenarios where all results are either true or false, such metrics have limited relevance when interpreting results.” I do not fully agree. For a power analysis at the design phase, typically both an effect size and the variance need to be estimated. In my opinion, a post-hoc power analysis could be relevant for interpreting a negative result, to take into account the observed variance. If the variance is a lot larger than a priori expected, then the study could have been underpowered. Perhaps adjust the text, or explain why I am wrong.

I do not understand “Finally, depending on the area of research overlooked, true positive results may be more harmful than false positive results”. Is meant that setting too stringent boundaries for significance can be harmful? Consider rephrasing to make its meaning clear.

Is the topic of the opinion article discussed accurately in the context of the current literature?

Partly

Are arguments sufficiently supported by evidence from the published literature?

Partly

Are all factual statements correct and adequately supported by citations?

Partly

Are the conclusions drawn balanced and justified on the basis of the presented arguments?

Partly

Reviewer Expertise:

Neuroscience

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Schmidt

Amand

University College London Faculty of Population Health Sciences, London, England, UK

Competing interests: No competing interests were disclosed.

4 2 2025

I have two comments on the first point.

Response: thank you, the following proof is provided in the updated supplementary material, also confirming this relationship is independent of sample size.

“

To describe the relationship between p-values and observed power we first let g represent the commulative density function of the p.d.f. f , that is

g t = -∞ t f x dx .

Next, say we observe test-statistic t o , in that case the p-value for a one-sided tests would simply be 1-g t o , and assuming f is symmetrical about zero the p-value for a two-sided test would simply be 2× 1-g t o .

If we let t c represent the critical value of a statistical test beyond which the null-hypothesis is rejected, power can be calculated as g t o -tc +g( -t 0 - t c ) , where in most cases either the first or the second term will be close to zero and can be ignored.

As an example, let f be the p.d.f. of the standard normal distribution, with t 0 =2 and t c =1.96 . In this case the two-sided p-value is 2× 1-g 2 ≈0.46 , and power would be g 2-1.96 +g -2-1.96 ≈0.52+0.00=0.52. Similarly, say t 0 =0 (i.e., the null-hypothesis is true), and f and t c are the same as before, we then find that 2× 1-g 0 =1 and g -1.96 +g -1.96 =0.05 . In other words when the null-hypothesis is true the p-value would be 1 and power would be 0.05 (or more generally power would equal to the area excluded by t c ).

As such there exists a one-to-one relationship between the p-value and observed power based on t o . Furthermore, and perhaps counterintuitively observed power is smallest when the p-value provides most support for the null-hypothesis and vice versa. This is a reflection of the p-value evaluating t o assuming the null-hypothesis is true, whereas observed power evaluates t o assuming this hypothesis is false. As in the previous section, these expressions do not depend on the amount of data collected and are therefore exact. Furthermore, given that a statistical test, comparing t o to t c , cannot be used to prove the null-hypothesis is true (as shown in the preceding section), the same holds for derived metrics including p-values and power. In fact power can only be calculated assuming the null-hypothesis is false, and hence cannot be used to argue in favour of the null-hypothesis being true.

“

Response: Thank you for raising this important point. The manuscript has been updated to clarify that the raised issues with observed power are of general concern irrespective of the whether the point estimates reflect the study results or some pre-existing notion of relevant difference. The concerns about power pertain to its use as a metric to interpret current results. In doing so the manuscript more clearly differentiates with power and/or sample size calculations to inform future research (which are without fault) and power calculations for already completed studies.

Page 6-7

“

As well as page 9

“

Second, while notions of power and type 1 errors are essential at the study design phase, because these deal in hypothetical scenarios where all results are either true or false, such metrics have limited relevance when interpreting results. Power and type 1 errors can be framed in terms of probabilities because the analysis has not yet been conducted. Once the experiment has been completed, these hypothetical probabilities are immaterial, and one is simply confronted with an unknown proportion of true-positive results.

“

Response: Thanks for raising these important points. The manuscript has been updated to clarify this involves a general testing procedure which does not make assumptions on the test statistics or the set of p-values following a normal distribution. Furthermore, the manuscript now explains that there may be a need to account for dependencies between sets of p-values.

The following has been included on page 8 “

“

And the figure 5 note “

N.b. The p-values were derived by arbitrarily sampling 1,000 test statistic from a normal distribution and leveraging its cumulative density function to calculate the area on the left and right side of the sampled test-statistic. Specifically, the employed standard distribution had based on random draws from a normal distribution with a a standard deviation of 1 and mean of either 0 or 2, when the null-hypothesis was true and false, respectively. Please note that the normal distribution is only used as an exemplar, and alternative distributions with a known cumulative density function (e.g. chi-square, beta, or gamma) could have been used instead.

“

These are my main comments. I also have a number of smaller remarks.

Response: This has now been integrated with the sections describing power, to further support the argumentation that (observed) power calculations are futile because the null-hypothesis can never be empirically shown to be true. As such this section is far from philosophical or redundant.

Response: The following clarification was added.

Page 7

“

Response: Thanks for pointing this out, the manuscript has been updated to refer to this as the false discovery rate, including a formal definition.

Page 7

“

An often overlooked point is that, depending on the unknown balance between false positives and true positives in a set of test results, applying multiplicity correction can sometimes increase the false discovery rate (i.e., the fraction of false positives divided by the total number of positive tests) instead of reducing it. For example, Figure 4A presents a naive expectation of multiple testing corrections, where the false discovery rate decreases from 1/3 to 0. However, there is no reason why the scenario depicted in Figure 4B may not occur; here, the false discovery rate increases from 1/3 to 1.

“

Given that the false discovery rate can only be defined based on the knowledge about which hypotheses are true and which are false, the second part of the question is impossible to answer for empirical research. However, the following thought experiment can provide some intuition. We know that the incidence of cardiovascular disease increases with age, which implies that in a sufficiently large study there would be a significant association between age and cardiovascular disease, say the p-value is 0.001. If we conduct a single test at an alpha (type 1 error rate) of 0.05 we would correctly reject a null hypothesis of no association. If we subsequently preformed four additional tests of association using completely random pairs of variables (i.e., where the null hypothesis is true), the Bonferroni multiplicity corrected alpha would be 0.01 and we would not reject the null hypothesis that age associates with cardiovascular disease. If however due to random chance one of the p-value for one of the remaining tests is smaller than 0.01 we would incorrectly reject the null-hypothesis thereby increasing the false discovery rate from 0 (when we only test for age) to 1/5.

Fig. 5: The number of samples used for producing the figure is missing.

Response: This was added.

“comparing the set of p-values for all the aforementioned outcomes against a uniform distribution resulted in a p-value of 0.02,” Which test is used for the comparison.

Response: This has been added.

“

Utilizing a non-parametric Kolmogorov-Smirnov test to compare the set of p-values for all the aforementioned outcomes against a uniform distribution nevertheless resulted in a p-value of 0.02, suggesting that the protective effect of rivaroxaban is shared across multiple cardiovascular outcomes.

“

Response: Thank you.

Response: The requested explanation has been included. Please refer to page 7-8.

“

Response: The following has been added to clarify this matter further.

Page 9

“

Finally, while decreasing the significance threshold (e.g. from 0.05 to 0.005) decreases the type 1 error rate this decreases power as well, and hence may decrease the number of true associations discovered. Depending on the area of research overlooked, true positive results may be more harmful than false positive results. For example, protein drug targets identified in early drug development are often subjected to a substantial number of follow-up analyses, which filter out false positive results. Such follow-up studies, however, rarely expand the number of candidates, hence suggesting a more inclusive approach might be more considerate. In settings more proximal to clinical implementation and less discovery oriented, such as phase 3 clinical trials, stringent multiple testing correction is clearly called for.

“

10.5256/f1000research.174018.r350489

Reviewer response for version 1

Cui

Ying

1 Referee https://orcid.org/0000-0002-3697-5155 1Stanford University, Stanford,, California, USA

Competing interests: No competing interests were disclosed.

2 1 2025

2025

recommendation

approve

The author have made very interesting points on common mistakes when interpreting results from statistical tests that fail to reject the null hypothesis. There are some minor comments as listed below:

1. In Page 5, it was mentioned that "Hence, a more relevant alternative to post-hoc power calculations is to evaluate the extent to which the CI includes clinically relevant effect estimates, which is in line with the aforementioned equivalence/non-inferiority approach.". It can be helpful if the author could provide a specific example (e.g. further discussion with the two HR estimates using equivalence/non-inferiority approach) to further illustrate this.

2. In the last line of Page 5, "naïve" should be "naive".

3. In the last lines of Page 7, it was recommended that "researchers should routinely apply composite null hypothesis tests evaluated against meaningful bounds of

insignificance". It would be helpful if the author could provide more discussion with suggestions on how to identify the "bounds of insignificance".

Is the topic of the opinion article discussed accurately in the context of the current literature?

Yes

Are arguments sufficiently supported by evidence from the published literature?

Yes

Are all factual statements correct and adequately supported by citations?

Yes

Are the conclusions drawn balanced and justified on the basis of the presented arguments?

Yes

Reviewer Expertise:

Hypothesis testing, Biostatistics

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Schmidt

Amand

University College London Faculty of Population Health Sciences, London, England, UK

Competing interests: No competing interests were disclosed.

4 2 2025

Reviewer Comments:

Response: Thank you, we have included the following example, using the VOYAGER PAD trial results.

Page 6

“

For example, the VOYAGER PAD HR estimate of 0.86 (95%CI 0.40;1.87) for bleeding risk in people with endovascular PAD clearly shows that the collected data supports a wide range of effect estimates, including potentially harmful associations. However, because the confidence interval only partially overlap with the proposed (hypothetical) upper bounds of acceptable harm of 1.25, testing against this bound results a p-value of 0.17 which is considerably smaller than testing against the complete absence of an effect: p-value 0.70. By comparison, the observed power estimate for these results is 7%, which implies that if the true HR was 0.86 one would have rejected the strict null-hypothesis in 7 out of 100 repeated experiment. As such observed power provide limit information relative to the presented alternative approaches, particularly the confidence interval based approach which allows for an informative discussion of benefits and harms in terms of effect magnitude(s).

“

Reviewer Comments:

2. In the last line of Page 5, "naïve" should be "naive".

Response: Thank you.

Reviewer Comments:

3. In the last lines of Page 7, it was recommended that "researchers should routinely apply composite null hypothesis tests evaluated against meaningful bounds of

insignificance". It would be helpful if the author could provide more discussion with suggestions on how to identify the "bounds of insignificance".

Response: The following has been included on page 4

“