Accumulation Bias in meta-analysis: the need to consider time in error control

The paper explains how bias arises in meta-analysis, as studies never are a random sample and the timing of a meta-analysis neither is random. Timing of the meta-analysis – if performed – and the results of the studies are obviously dependent, resulting in accumulation bias and inflated type I errors. The parallel between conducting studies and betting while continuously reinvesting the profits is an intriguing one.

I apologize for the long duration of my peer review: the paper was rather intense and long, and sometimes a bit difficult to follow, due to the condensed writing style. It might benefit from some extra sentences and some additional, concrete examples. And, to my opinion the content might also be suitable for a few papers.

However, I could find only minor editorial issues, and I congratulate the authors with this well-written, very relevant paper.
 
Page 1, abstract
It feels a bit like a contradiction when is stated (halfway):  “…, no valid p-value test can be constructed. Second, tests based on likelihood ratios withstand Acc. Bias: they provide bounds on error probabilities that remain valid despite the bias.” Also last paragraph: Taking up likelihood ratios… allows for valid tests.

Probably it has to do with the term p-value based statistical tests that I am not familiar with (as opposed to likelihood ratio tests). I think it has to do with the explanation in the discussion, that (p-value) tail area approaches to testing are very sensitive to the shape of the sampling distribution, but in the abstract, this was not clear to me.
 
1. Introduction
I like how you start with the two quotes.
First column at the end: knowledge and all decisionS made along the way. (s is missing)
 
2. Accumulation Bias
Somewhat difficult (although correct) sentence: The crucial point is that not all pilot studies or small study series will reach a meaningful size and that doing so might depend on results in the series.
One but last sentence: So meta-analysis also report….  (should be meta-analyses)
 
Section 3.6
Typo: The inflation actual inflation in the type-I error…
Figure 1: in black and white print the colours in the legend seem to differ from the colours in the graph.
Table 2: If possible, it might be handy to add the P̃₀ and P₀ to the title (after bias only and after bias as well as impaired sampling distribution).
 
Section 4.5
Typo: These cumulative meta-analysis judge…. : should be meta-analyses
 
Section 5
Here you refer to “another illustration”… as the Toy Story Scenario. However, where do you discuss this scenario?
 
Section 6.1
Typo: “But this effects…”. Also somewhat unclear sentence. I suppose that you mean that is difficult to define a cut-off for the number of early studies to be excluded from meta-analysis.
 
Section 6.2
Here you compare the prior odds of Ioannidis with the fraction of pilot studies from the null and alternative distribution π / (1-π). However, you did not define π before, and if I understand it correctly, π is the fraction of studies from the alternative distribution, although this text (first line) suggests the other way around.
 
Section 7.1
Formula 7.2: Please define H₁ and H₀.
Please edit sentence directly below formula: …16 times as many true rejections than (as?) false rejections (with?) γ = (1-β) / α.
Formula 7.4: P₀ : should be integral over z₁, …, z_t (instead of z₂) (twice). Should it not contain also dz(1).…dz(t)?

I noticed that you don’t use the word test, only “error control”. It is not fully clear to me: if we use the threshold based on the Bayes posterior odds, does that also result in a p-value, or is it just a yes/no answer? Or can we use a distribution? (you elaborate on this only in Section 9).
 
And how do we specify R= π / (1-π)?  Should this be influenced by the study results seen thus far? As you state in section 6.2 on π / (1-π): “the fraction of pilot studies from the null and alternative distribution. … The only available source of information would be previous study results… “. However, this would mean that – indeed -  depending on the timing of the meta-analysis, we would define a different R.
Or should we use the same – more general -  R as Ioannidis, ie 1:1, or 2:1?
 
Interesting is also that the threshold, that is based on the pre-experimental rejection odds, becomes more stringent if we believe the ratio of true to false rejections to be higher, e.g. if R =2 and γ = 16, the threshold becomes 32, but if R=1, the threshold is 16. Could you elaborate on that? You do elaborate a bit in Section 8, but for me it is still not very clear.
 
Section 9
Edit, 2^nd paragraph, … a series of bets s(Z₁), … against the null hypothesis that make a profit….
I suggest to (re)move “against the null hypothesis”  to facilitate easier reading.
Typo: each null result might costs 1 dollar (should be: cost)
“If you cash out, your profit is s(Z₁) s(Z₂) s(Z₃)”. Are the s(Z) bets not odds or probabilities?  Should we not add the profit here? And subtract the 1 dollar initial investment? Or do I show here my lack of knowledge on gambling?  Only later, when you suggest s(Z) = f₁/f₀, this makes more sense.
Typo, second column before quote of Shafer: twice “under the null”.
Inspecting the betting profit of a study: do you mean calculating the LR for that study?
I don’t understand the following sentence: “The LR has the ability to maximize the rate of growth among all studies in a series”.
 
Section 10
Typo: 3^rd paragraph: separate meta-analysis of various sizes…, should be meta-analyses