What is reproducibility?

This is a thoughtful piece that attempts to offer a construct that will help define research reproducibility. They say that their purpose is to offer an operational definition of reproducibility that they claim a previous paper entitled “What does research reproducibility mean?" (with myself as first author) did not address:
 
“Goodman et al. did define three types of reproducibility (methods, results, and inferences) and stated that confusion arises when, inadvertently, people use reproducibility as a synonym for “truth”⁶ . We read their paper as being about truth although its title suggests otherwise.”
 
The claim that the 2016 paper ¹ was about truth and not reproducibility is not quite right. Let us see exactly what the prior paper said:
 
Results reproducibility (previously described as replicability) refers to obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible. … this might be clear in principle but is operationally elusive. The problem arises in settings where there is substantial random error in any result, making unclear the criteria for considering results to be “the same.” The intuition and logic of results reproducibility are derived from systems that are deterministic or for which the signal-to error ratio is exceedingly high. But, when the same intuition and logic are applied to studies with substantive stochastic components, the paradigm of accumulating evidence might be more appropriate than any binary criteria for successful or unsuccessful replication.
….
Statistical significance by itself tells very little about whether one study has “replicated” the results of another. For example, two studies that show identical 10% survival differences between the treatment and control arms would have very different degrees of statistical significance if their sample sizes were substantially different. If one was highly significant and the other far from significance, the two studies might be reported individually as supporting opposite conclusions, in spite of the fact that they are mutually corroborative. An interpretive error complementary to the one described above involves the assumption that multiple studies that fail to demonstrate statistical significance necessarily confirm the absence of an effect.
…..
It is easier to statistically define non-replication than replication, through statistical tests of heterogeneity, which can evaluate whether the difference between two or more experimental results might be due to the play of chance. Two or more studies are judged to be statistically heterogeneous when the between-study variance in reported effects is substantially greater than what is expected from sampling error. Such tests, however, are greatly underpowered and therefore unreliable when comparing several studies, particularly when they are small or imprecise (17). Conversely, when there are many large studies, tests for heterogeneity might demonstrate statistical heterogeneity (and, therefore, lack of results reproducibility) even if the effect sizes of different studies are close (17) and regarded as scientifically equivalent. Therefore, a preferred way to assess the evidential meaning of two or more results with substantive stochastic variability is to evaluate the cumulative evidence they provide vis-á-vis a hypothesis of interest and not whether one contradicts or discredits the other through the lens of statistical significance.
 ---------------------------------------------------
So it should be apparent that the 2016 paper does indeed address exactly what this paper addresses, including the notion that results can differ in statistical significance yet be regarded as "scientifically equivalent”. That is essentially what this paper goes on to try to define, with a "zone of equivalence” that defines "scientifically equivalence”. But as these authors acknowledge in the legend of Figure 1, “…. even after delta and the type of confidence limit have been chosen, uncertainty may persist if confidence limits overlap the boundaries of delta.” The problem is that the confidence intervals will quite often cross the boundaries of delta, and so we are left with the same conundrum that the original paper said was inescapable.

The point of the original paper was that trying to define “reproducibility” was in the end not very constructive, and that if we turned our attention instead to the cumulative evidence represented by several studies, instead of whether they "reproduced" or not, we could avoid these distinctions, which ultimately serve little purpose. The authors here are right that I believe that the goal of science, and of scientific studies, is to move us closer to the truth. I contend that debates about which and how many studies reproduced do not, particularly when that definition is elusive. The prior paper did indeed tell us that convergence on the truth should be our lodestar, not an arbitrarily defined reproducibility criterion, which even with the improved version offered here does not provide a clear verdict in the vast majority of cases.
 
I agree with the authors that it is helpful to have some notion of differences that make a difference, and thereby scientific equivalence. But the degree of imprecision in most health studies precludes an unambiguous conception of reproducibility even if one introduces that interval. I also agree with the authors’ conclusion that “….the concept of reproducibility (repeatability, precision) should be distinguished from validity (“truth”)”, but disagree that the purpose of assessing reproducibility is anything other than getting at the truth, and still believe that the cumulative evidence model and not the reproducibility model - which cannot be clearly defined - is what gets us there.