This article is necessarily limited in scope as it presents results for just one trial sponsor, and we do not know if that sponsor is representative. However, as the authors are from just one institution, it is of course perfectly reasonable that they have focused on their own institution, so this is not intended as a criticism, merely as an observation.

Coens et al have done a good job of presenting a more detailed analysis of the studies from their institution, and have shown convincingly that the estimate of their publication rate from the automated Trial Tracker was substantially inaccurate for their institution, by means of the gold standard of a manual search. They provide a sensible and balanced discussion of the limitations of the automated search algorithm more generally, pointing to some possible unintended consequences. While those unintended consequences are at this stage purely speculative, it does no harm to bear in mind what the risks are of an automated process such as the Trials Tracker.

I have 2 suggestions for improving the paper. First, Coen et al state that only 29 of the 38 trials identified by the Trials tracker were "eligible studies", which they define as "completed since 01/01/2006, interventional phase II or III, led by EORTC". When I applied those criteria myself to the EORTC trials identified by the Trials Tracker, I found 30 eligible studies. As far as I could tell from the Trials Tracker data, all 38 studies were completed since 01/01/2016 and were led by EORTC, and 8 studies were not drug interventions. It would be helpful if Coen et al could be more explicit about why they excluded 9 trials from their analysis.

Second, I think the finding that some trials were not identified as published by the Trials Tracker despite a publication that was clearly linked in the clinicaltrials.gov record deserves more emphasis. Although this is mentioned in the paper, a casual reader might miss it, and this is perhaps the most important finding in terms of a way in which the Trial Tracker could easily be improved.

Reviewer Expertise:

NA

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

This paper from Coens et al discusses an important aspect of the recently published TrialsTracker database/analysis of the EBM Data Lab (University of Oxford). The paper introducing it from Drs. Powell-Smith and Goldacre ¹ initiated exciting comments ¹ , press statements ² and even tweet exchanges with pharmaceutical companies ³ , raising similar concerns; the present article is the first however to formalize such criticisms through the detailed analysis of a particular institute's trials.

I welcome this investigation as a systematic substantiation of the concerns raised in the aforementioned sources. Albeit pertaining to a single institute, the results are at least illustrative – even if not representative – for the entire TrialsTracker project. (A notable – and important – exception that is not discussed in the present manuscript is the question of results posted to company websites.)

The presentation of the findings from Coens et al is almost flawless in my opinion, with the following minor remarks:

I don't see how 38 changed to 29 (number of relevant EORTC studies). The said criteria – completed, has completion_date and it is later than 1 Jan 2006, interventional, phase 2 or 3 – results indeed in 38 records. Yet, Coens et al reports only 29, saying that these are "relevant", but how they define relevance (ie. what 9 studies were excluded and why) is not discussed at all.

My major comments are therefore rather about missing details and potential further improvements:

TrialsTracker looks up results from clinicaltrials.gov (results section) and Pubmed (only through NCT ID as SI). While non-publication in Pubmed might have several – not necessarily malicious or negligent – reasons, such as the publication being rejected or a long review process, clinicaltrials.gov has no such limitation, so non-disclosure there seems to be much more inexcusable at first glance. Coens et al touches this issue, but only extremely briefly, stating that "[uploading results to clinicaltrials.gov] presents a substantial administrative burden, and does not allow results to be put into context''. I'd really welcome a more detailed discussion of the first part: how large is this burden, is it in fact prohibitive...? (Especially for organizations with tens of thousands of employees and clinical trials with a budget in excess to ten million US dollars.) As far as the second part is concerned, I disagree: the aim of the deposition of results in repositories is not its presentation "in context", but simply making them available. Not that the presentation of the context is not important, but it is a separate issue. (Actually, availability of raw results might even be beneficial, avoiding potential biases introduced by a biased context ⁴ .) Thus, this sentence of the authors should be elaborated in more detail.

Finally, let me note that the authors – quite rightly – summarize the drawbacks of the automation, but to be balanced, its strengths should have been mentioned: first, that the automated nature allows investigation on a scale in quantity that cannot be achieved – or only through extreme measures – with manual checking, and second, perhaps even more importantly, that the automated algorithm is totally transparent and surely free of any subjective decisions. Finally, even if the algorithm is imperfect, at least it is uniformly imperfect, thus the results are likely comparable in spite of this. As far as the distribution of these false results is similar among sponsors, their TrialsTracker score can still be compared. (That's the reason why one cannot simply correct EORTC's results, for example based on this paper, because that would mean that even this comparability is lost.)

Reviewer Expertise:

NA

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.