Retract p < 0.005 and propose using JASP, instead

Seeking to address the lack of research reproducibility in science, including psychology and the life sciences, a pragmatic solution has been raised recently: to use a stricter p < 0.005 standard for statistical significance when claiming evidence of new discoveries. Notwithstanding its potential impact, the proposal has motivated a large mass of authors to dispute it from different philosophical and methodological angles. This article reflects on the original argument and the consequent counterarguments, and concludes with a simpler and better-suited alternative that the authors of the proposal knew about and, perhaps, should have made from their Jeffresian perspective: to use a Bayes factors analysis in parallel (e.g., via JASP) in order to learn more about frequentist error statistics and about Bayesian prior and posterior beliefs without having to mix inconsistent research philosophies.


Argument
Seeking to address the lack of research reproducibility due to the high rate of false positives in the literature, Benjamin et al. (2017a); Benjamin et al. (2017b) propose a pragmatic solution which "aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance" (also Savehn, 2017): to use a stricter p < 0.005 standard for statistical significance when claiming evidence of new discoveries.
The proposal is subject to several constrains in its application: (1) to claims of discovery of new effects (thus, not necessarily to replication studies); (2) when using null hypothesis significance testing (arguably Fisher's approach, perhaps even Neyman-Pearson's, but excluding other p-value-generating approaches such as resampling); (3) in fields with too flexible standards (namely 5% or above); (4) when the prior odds of alternative-to-null hypothesis is in the range 1-to-5, to 1-to-40 (stricter standards are required with lower odds); (5) for researcher's consumption (thus, not a standard for journal rejection, although "journals can help transition to the new statistical significance threshold"; also, "journals editors and funding institutions could easily enforce the proposal", Wagenmakers, 2017; and "its implementation only requires journal editors to agree on the new threshold", Machery, 2017); (6) while still keeping findings with probability up to 5% as suggestive (and meriting publication if so "properly labelled"); (7) despite many of the proponents believing that the proposal is nonsense, anyway (that is, it is a quick fix, not a credible one; also Ioannidis in Easwaran, 2017;Resnick, 2017;Wagenmakers, 2017;Wagenmakers & Gronau, 2017).
The main analyses supporting the proposal were the following: (a) a calibration of p-values to Bayes factors under certain plausible alternatives (their Figure 1, Benjamin et al., 2017b, p. 2); (b) an estimated false positive rate under certain plausible prior odds of alternative-to-null hypotheses as a function of power (their Figure 2, Benjamin et al., 2017b, p. 3); and (c) a "critical mass of researchers now endors [ing] this change" (a.k.a., the 72 co-authors). Meanwhile, the proposal downplays the costs due to increasing sample size in order to ensure enough research power, the potential increase in false negatives and misconduct, and the existence of alternative solutions well known by, at least, some of the authors (despite Machery's, 2017, claim to the contrary).
Notwithstanding its potential impact, Benjamin et al.'s (2017a);Benjamin et al.'s (2017b) proposal is not recognizant of the entire engine of science (Figure 1), which has motivated an even larger mass of authors to offer their counter-arguments.

Amendments from Version 1
Minor changes incorporating reviewers' recommendations: • [1] The legend in Figure 1 now defines the acronyms in the figure.
• [2] A new reference to Perezgonzalez (2015) now implies that the pseudoscientific label attached to the NHST element ( Figure 1) follows from the rhetoric in such reference.
• [3] A second note clarifies that JASP also allows to use Cauchy, Normal and t-distributions as informed priors for the alternative hypothesis.
• [4] There is a new acknowledgement section.

Counter-arguments
From Philosophy of science, Mayo (2017a), in particular, provides important counter-arguments against the philosophical standing of Benjamin et al. 's proposal. Quoting Greenland et al. (2016), Mayo asserts that whether p-values exaggerate the evidence against the null hypothesis depends on the philosophical background within which such claim is made: It may seem so from a Bayesian perspective, but both from a frequentist and an error statistics perspective, it is the Bayes factor which exaggerates the evidence (also 2017e). "I find it especially troubling"-she continues-"to spoze an error statistician…ought to use a Bayes Factor as the future gold standard for measuring his error statistical tool…even though Bayes Factors don't control or measure error probabilities" (2017b). Furthermore, Mayo pinpoints the (old) fallacy of transposing the conditional, whereby the (error) probability of a test is confused with the (posterior) probability of a belief (also Trafimow et al., 2017). And despite "60 years (sic) old…demonstrations [showing] that with reasonable tests and reasonable prior probabilities, the disparity vanishes…they still mean different things" (2017c). In particular, "in an adequate account [of severity testing], the improbability of a claim must be distinguished from its having been poorly tested. (You need to be able to say things like, 'it's plausible, but that's a lousy test of it.')" (2017d). And "the method for such checking is significance tests!" (2017b). The pseudoscientific NHST element (as conceptualized in Perezgonzalez, 2015) is the cornerstone of Benjamin et al.'s proposal, as it mixes Fisher's tests of significance with Neyman-Pearson's alternative hypotheses, Type II errors and power estimation, and with Bayesian prior probabilities, in order to argue about the false positive rate as the culprit for the lack of replicability. Indeed, the false positive rate argument has been heavily criticized by several authors: Mayo (2017b; also Byrd, 2017) chiefly derides as questionable the prior probabilities given to the hypotheses; Colquhoun (2017) denounces the credibility of Benjamin et al.'s false positive rate calculation for some of the prior probabilities used, which he sees as a rather liberal rate; Chapman (2017; also Byrd, 2017) claims that the prior probability ratios for the hypotheses are not realistic; Llewelyn (2017) raises an argument on prior probabilities and their combination with accumulated evidence in order to estimate the probability of future replication, not finding a more restrictive level of significance an adequate antecedent for increasing such probability; Krueger (2017) adds that, under the new proposal, the proportion of false negatives rises faster than the speed at which false positives drops, leading Phaf (2017) to wonder whether, because of it, "it makes more sense to increase, rather than reduce, the significance level for novel findings"; Kong (2017) states that the false positive rate formula is misleading and finds the expected false positive rate when testing the conventional 5% threshold.
Furthermore, Perezgonzalez (2017) (2017) argues, instead, for the need to be humbler and not generalize to populations, irrespective of statistical significance, effects that may only be present in the sample.
The descriptive data analysis element was briefly touched upon by Mayo (2017a)-who recommends abandoning significance testing in favour of inferring the magnitudes that are well (or poorly) indicated by the data (a.k.a., CIs)-and Lew (2017) The hypothesis testing element (namely full Bayesian hypothesis testing) has been seldom touched upon but briefly by Llewelyn (2017), who extended the need for prior probabilities and even more accumulated evidence as pre-requisite for estimating the probability of hypotheses being 'true'.
In conclusion, despite Benjamin et al.'s best intentions, their proposal only reinforces the NHST pseudoscientific approach with a stricter bright-line threshold for claiming evidence it cannot possibly claim-irrespective of how well it calibrates with Bayes factors under certain conditions-while dismissing potential consequences such as the stifling of research and its publication, an increase in false negatives, an increase in misbehaviour, and a continuation of pseudoscientific practices. Furthermore, theirs is a frequentist solution many of their Bayesian proponents do not even believe in, born out of a false belief of lacking equally simple but better-suited alternatives (Machery, 2017).
In reality, an equally simple and better suited alternative exists, perhaps the only one the authors could be entitled to make from their Jeffresian perspective, hence our own recommendation: Use JASP, the stand-alone, free-to-download, R-based statistical software with user-friendly interface, developed by Wagenmakers' team (https://jasp-stats.org 1 ). JASP allows for analysing the same dataset using frequentist (Fisher's tests of significance) and Bayesian tools (Jeffreys's Bayes factors 2 ) without necessarily forcing a mishmash of philosophies of science (e.g., see Supplementary File 1). JASP allows for the Popperian (also Meehl's, Mayo's) approach to severely testing a null hypothesis but also to ascertain the credibility of tests based on the observed data. JASP allows researchers to qualitatively calibrate their significance tests to Bayes factors but also the possibility of calibrating Bayes factors to frequentist tests so as to ascertain their error probabilities. JASP is an existing simple and easy application that shows two interpretations of the same data, which aligns well with the training undertaken by frequentist and Jeffreysian researchers alike while allowing them the opportunity to learn the alternative philosophy-irrespective of which they will choose for publication-and may as well help diminish the misinterpretation of results and the reaching of unsupported conclusions. In so doing, it may even help improve replicability.

Competing interests
No competing interests were disclosed. I consider this is a valid contribution to the debate on this topic but like many such contributions it reflects a particular viewpoint, my own contributions in this field are no exception, and like many contributors, and, again, I am probably also guilty, there is minimal recognition that their own views are not reasonable from other perspectives. In particular (see below) they are over-fond of the adjective and pseudoscientific their use of it without further justification is, in my view, a form of pseudo-argumentation.

Grant information
Particular objections that I have are the following.
They state: "The pseudoscientific is the cornerstone of Benjamin 's proposal, NHST element et al. as it mixes Fisher's tests of significance with Neyman-Pearson's alternative hypotheses." This, in my opinion, is an unsubstantiated and misleading jibe. First, although this claim is often made, it is not correct that NHST is some sort of monstrous hybrid of Neyman-Pearson (N-P) and Fisherian views. It very naturally uses common elements that belong in both.
(2) For instance, Fisher, although usually associated with P-values, introduced the habit of tabulating percentage points in . Secondly, as Lehmann, a classic proponent of the N-P Statistical Methods for Research Workers view explained in , P-values are a practical way of allowing scientist Testing Statistical Hypotheses who may not agree on appropriate Type I error rates to share results efficiently. Furthermore, although I have little enthusiasm for what Benjamin et al are doing, I think that fact that NHST seems to represent some sort of a fusion of Fisher and NP (a point that is much exaggerated and of little relevance) is the least important aspect of what they are doing. If they had rejected everything the Fisher proposed and described what they were doing as trying to convince diehard N-P acolytes that the common 5% level was better changed to 0.5%, it would have zero technical effect on what they are proposing and would not make it either better or worse. Whatever problems or virtues there are with the Benjamin et al proposal, the Fisher, N-P fusion is a complete red herring. Will JASP and Bayes Factors (BF) really save statistical inference? I doubt it. One of the problems with the Jeffreys approach is the huge influence that the lump of probability on the 'null' has on the posterior inference. NHST is far less dependent on this and, in a context in which I work, that of drug development, the practical challenge is to avoid recommending a treatment that is worse. it is a strength of NHST that one does not have to worry too much about whether one is talking about . Of course, Bayesians precise(or point hypotheses) on the one hand or dividing ones on the other could regard this as not a strength but a weakness, on the lines that if it make a difference to does Bayesian posterior statements ( ) it to in the NHST and the difference can be enormous ought 1,2 1 2 1 3.
Bayesian posterior statements ( ) it to in the NHST and the difference can be enormous ought context. (It is a very common habit of Bayesians to abrogate the right of judging other systems by theirs.) However, I also note that amongst the Bayesian commentators on the recent ASA statement, some dismissed precise hypotheses as being completely irrelevant. Like many commentators, they appear to accept much of the replication crisis at face value. However, there are four aspects here that are important. i) Gaming can effect any system (as they recognise) ii) Much of the discussion has taken it as obvious that failure to replicate is a problem of the original study , rather than a shared problem. However, it can only be the former if the only second study is perfect and that includes not being subject to sampling variation itself. What thus becomes important is to judge how well studies of infinite size are predicted, not those that just happen to be the same size as the original (2). iii) It assumes that what is important is the extent to which the P-value predicts the P-value but there is no claim under NHST that this is so. What is relevant is the extent to which it predicts the sign of the true difference Furthermore, to shackle the Bayesian approach with the same standard (what's sauce for the goose is sauce for the gander) would require that a high BF predicted a further high BF (using data from the subsequent study ) with high probability. JASP on its own will not deliver this. iv) Finally, if the only replication that only matters is failure to replicate 'significance' then the proposal of Benjamin et al is nowhere near radical enough. 'Significance' must never be granted. On the other hand, if false negatives are also a problem, and not just false positives, then the solution has to be rethought from the beginning. (To be fair to the authors, they do cite Krueger's, 2017 warning about this but it is unclear to me to what extent they take it on board in what they are proposing.) However, putting aside my objections to some of the polemics of the article, the idea of supplementing P-values by other inferential statistics strikes my as being sensible and I certainly approve their suggestion of keeping Bayes factors separate from P-values. In fact, I have no objection to Bayes Factors provided that their very tentative nature is recognised and I certainly sign up to the idea of the utility of having different ways of looking at data . Of course, this doesn't mean that the double analysis is the quick fix-all solution, though. I think Gelman (or is it Senn?) has also been wondering why when we could go for a full Bayesian analysis instead. I also BF understand Mayo's point focused on error statistics. Yet all those perspective are focusing on the problem from different points-of-views, and such points-of-view need to be understood in order to make sense of each claim.

Discuss this Article
In a nutshell, next time I go to the doctor for a test, I want that test to be pretty good at reducing false positives (Fisher). I also want it to be pretty good at reducing false negatives (Neyman-Pearson; BF The problem with likelihood ratios is that their use could result in the execution of an innocent person. See http://www.dcscience.net/2016/03/22/statistics-and-the-law-the-prosecutors-fallacy/ (in particular, the island problem).
I take it that the main aim of a statistical test is to avoid making a fool of yourself by claiming that an effect exists when in fact your observation is just chance. The probability of this happening is the false positive exists when in fact your observation is just chance. The probability of this happening is the false positive As you have explained, a large proportion of researchers still think that this is what the p-value risk (FPR). tells you, quite wrongly, The problem lies in the fact that, in order to calculate the FPR you need to know something about the prior probability that the effect is real (as well as the likelihood ratio). Such information is very rare, and people will disagree about it.
The best way to circumvent this dilemma seems to me to follow a suggestion of Robert Matthews and to calculate the prior probability that would be needed to reduce your FPR to, say, 5% (that's what so many people think, mistakenly, that p = 0.05 achieves). The calculations are described in http://rsos.royalsocietypublishing.org/content/4/12/171085 and they can be done without using our R scripts using our web calculator, at http://fpr-calc.ucl.ac.uk/ (but please check the Notes tab before using it, if you haven't read the paper).
For example, one finds that, for a well-powered experiment in which you observe p = 0.05, you'd have to be 87% sure that there was a real effect before you did the experiment in order to achieve an FPR of 5%. It would be up to you to persuade editors and readers that this was a reasonable assumption. Clearly it isn't.
If you observe p = 0.005 then you'd need a prior probability of 0.4 in order to achieve an FPR of 5%. Again it is up to you to persuade people that it's a reasonable assumption. It might well be deemed reasonable for a plausible hypothesis, but if you were testing a homeopathic pill, it would be absurdly high.
The likelihood ratio can be regarded as the odds on there being a real effect when the prior odds are 1, i.e. a prior probability of 0.5. Since it is hardly ever justifiable to assume a prior larger than this. the likelihood ratio gives a measure of the minimum FPR -that for a plausible hypothesis. For example, if you observe p = 0.05, then the minimum FPR is 27%. If you observe p = 0.005, the minimum FPR is 3.4%.
But if the hypothesis were implausible, with a prior probability of 0.1, then p = 0.05 would correspond to an FPR of 76% -disastrously high. And if you observed p = 0.005, the FPR would still be 24%. Even p = 0.001 gives an FPR greater than 5% in this case: to achieve an FPR of 5% you'd need to observe p = 0.00045.
So I would conclude that, awkward though it is, it is asking for trouble to rely on likelihood ratios alone. You simply can't ignore prior probabilities just because you don't know them. Hence my suggestion to use the reverse Bayes approach, i.e. to calculate the prior needed to achieve an acceptable FPR. Of course people may disagree about how plausible this prior is. Sadly there is no way to do inductive inference in an entirely objective way, e.g, https://aeon.co/essays/it-s-time-for-science-to-abandon-the-term-statistically-significant Even if you are willing to believe that the chance your hypothesis is right, is as high as 50:50, so likelihood ratios would imply the FPR, it still seems better to express the result as an FPR, because probabilities are a much more familiar measure than likelihood ratios.
Only having written about the topic myself Competing Interests: