Comment on the “ TrialsTracker : Automated ongoing monitoring of failure to share clinical trial results by all major companies and research institutions ”

The purpose of this correspondence is to discuss the TrialsTracker, presented by Powell-Smith and Goldacre in their article ‘TrialsTracker: Automated ongoing monitoring of failure to share clinical trial results by all major companies and research institutions’ (2016) as a tool to discover publication bias in clinical trial results. The findings from one specific organization (European Organization for Research and Treatment of Cancer; EORTC) are compared with the actual publication history of the trials in question. We also present shortcomings of the method being used and suggestions for improvement to the proposed algorithm.

We read with great interest the article by Drs Powell-Smith and Goldacre 1 on the incomplete reporting of clinical trial results by pharmaceutical companies and research institutions.The necessity to publish results of all clinical trials, regardless of the trial outcome, cannot be denied 2 .Failure to do so is unethical not only towards patients who have participated in these trials but also towards the medical community at large which relies on unbiased reporting to make informed decisions both in clinical practice and research 3 .
The European Organization for Research and Treatment of Cancer (EORTC), as a non-profit pan European clinical research organization, very much supports this view.EORTC is driven by the mission to improve the survival and quality of life of cancer patients, and adheres to a strict policy to publish all of its completed trials in full 4 .We were therefore surprised by the results from TrialsTracker 5 stating that 52.6% of EORTC trials are missing results (20 trials out of 38).We downloaded the full trial dataset used by the tracker via GitHub (https://github.com/ebmdatalab/trialstracker).After selection according to the set criteria (ie. completed since 01/01/2006, interventional phase II or III, led by EORTC) a total of 29 relevant trials were found.The tracker classified these as: 14 with successful result reporting and 15 (51.7%) without results.We identified the latter 15 trials through the NCT ID number and cross-referenced this with the EORTC internal bibliography list.This (manual) investigation yielded the following results (see Table 1): -A total of 9 trials had been successfully published, but the NCT ID number was not listed in PubMed's Secondary Source ID field.For all of these trials the NCT ID was stated in the article itself and a link to the correct reference was provided in the publications section of ClinicalTrials.gov-Three further trials had been recently successfully published without mention of the NCT ID number.The reference was not yet present in ClinicalTrials.govbut was scheduled to be updated soon.
-The last three trials were still undergoing analysis, and the planned publications were in various stages of development.
This would put the EORTC under-reporting "score" at 3/29 or about 10% of its trials being overdue for publication.
Our investigation revealed several shortcomings of the automated tracker algorithm: -The decision to only accept results posted directly in ClinicalTrials.govor with a listed NCT ID in PubMed's Secondary Source ID field is very restrictive.EORTC does not post results in ClinicalTrials.govdirectly as this presents a substantial administrative burden, and does not allow results to be put into context.Other organizations may be in the same situation.
-The authors state that since 2005 "all major medical journals (through the International Committee of Medical Journal Editors) have required trials to be registered, and all trials should include their registry ID in the text."The majority of our trials, as identified by the tracker, fulfill these criteria, yet several were incorrectly classified due to absence of a specific PubMed field provided by the medical journals.Despite this omission, these trial results could be correctly found through recognized databases such as ClinicalTrials.govand PubMed or even a standard search engine like Google.
-For at least two studies, the NCT ID PubMed link was available for a publication that did not contain the actual study results.The EORTC 55971 study on neoadjuvant treatment in ovarian cancer was published in NEJM in 2010 6 .Yet this publication was not identified by the tracker, but two subsequent publications on exploratory subgroup analyses were considered as evidence of trial results publication 7,8 .
We also want to introduce two caveats to take into account when refining a tracker such as the one proposed: -The algorithm can be easily manipulated to inflate the success rate for any trial sponsor by either not listing trials as completed or by listing them as terminated in the registry.
-Also, once the trial is completed, any publication with an adequate NCT ID PubMed link is sufficient for the TrialsTracker algorithm, which means articles on quality assurance, subgroups, translational research, prognostic models, or other data not containing actual trial results will inflate the statistics.
As a general observation, we feel that tracking publications linked to trials without checking these publications for accuracy and adequacy represents a simplistic measure of publication reporting.A substantial source of bias lies in the incorrect publication of trial results, often done with the intention to present larger treatment effects 9 .We feel such a tracking system, by increasing pressure to publish all trials on short notice, may contribute to the problem by leading to compromises on the quality of the publication.
A straightforward approach to resolve this could be to add to clinical trial registries an indicator on the publication status of final trial results.The sponsor would be responsible for updating this indicator and for providing the actual reference.Registry administrators could then check the appropriateness of the reference based on criteria already required to check online posting of results, therefore providing independent confirmation that the trial results are adequately published.Such an indicator would allow for more accurate reporting and could be used to set up an automatic alert system.
The EORTC welcomes initiatives to improve clinical trial reporting.The EORTC has an explicit data sharing policy (http://www.eortc.org/investigators/data-sharing/)that allows anyone to request direct access to clinical trial data from completed studies.In Coens have done a good job of presenting a more detailed analysis of the studies from their et al institution, and have shown convincingly that the estimate of their publication rate from the automated Trial Tracker was substantially inaccurate for their institution, by means of the gold standard of a manual search.They provide a sensible and balanced discussion of the limitations of the automated search algorithm more generally, pointing to some possible unintended consequences.While those unintended consequences are at this stage purely speculative, it does no harm to bear in mind what the risks are of an automated process such as the Trials Tracker.
I have 2 suggestions for improving the paper.First, Coen state that only 29 of the 38 trials identified et al by the Trials tracker were "eligible studies", which they define as "completed since 01/01/2006, interventional phase II or III, led by EORTC".When I applied those criteria myself to the EORTC trials identified by the Trials Tracker, I found 30 eligible studies.As far as I could tell from the Trials Tracker data, all 38 studies were completed since 01/01/2016 and were led by EORTC, and 8 studies were not drug interventions.It would be helpful if Coen could be more explicit about why they excluded 9 trials et al from their analysis.
Second, I think the finding that some trials were not identified as published by the Trials Tracker despite a publication that was clearly linked in the clinicaltrials.govrecord deserves more emphasis.Although this is mentioned in the paper, a casual reader might miss it, and this is perhaps the most important finding in terms of a way in which the Trial Tracker could easily be improved.Powell-Smith and Goldacre initiated exciting comments , press statements and even tweet exchanges with pharmaceutical companies , raising similar concerns; the present article is the first however to formalize such criticisms through the detailed analysis of a particular institute's trials.

I have previously blogged about the Trials
I welcome this investigation as a systematic substantiation of the concerns raised in the aforementioned sources.Albeit pertaining to a single institute, the results are at least illustrative -even if not representative -for the entire TrialsTracker project.(A notable -and important -exception that is not discussed in the present manuscript is the question of results posted to company websites.)  1 should be improved by clearly marking which trial belongs to which category: has results or not, in the latter case the reason from the three listed (NCT ID not in SI field, no NCT ID given in the publication, really not published).In the current form it is difficult to match the concrete trials to the authors' statements (e.g.what are the trials that have a publication but not NCT ID given?).
My major comments are therefore rather about missing details and potential further improvements: TrialsTracker looks up results from clinicaltrials.gov(results section) and Pubmed (only through NCT ID as SI).While non-publication in Pubmed might have several -not necessarily malicious or negligent -reasons, such as the publication being rejected or a long review process, clinicaltrials.govhas no such limitation, so non-disclosure there seems to be much more inexcusable at first glance.Coens touches this issue, but only extremely briefly, stating that et al "[uploading results to clinicaltrials.gov]presents a substantial administrative burden, and does not allow results to be put into context''.I'd really welcome a more detailed discussion of the first part: how large is this burden, is it in fact prohibitive...? (Especially for organizations with tens of thousands of employees and clinical trials with a budget in excess to ten million US dollars.)As far as the second part is concerned, I disagree: the aim of the deposition of results in repositories is 3.

4.
as the second part is concerned, I disagree: the aim of the deposition of results in repositories is not its presentation "in context", but simply making them available.Not that the presentation of the context is not important, but it is a issue.(Actually, availability of raw results might even separate be beneficial, avoiding potential biases introduced by a biased context .)Thus, this sentence of the authors should be elaborated in more detail.

Coens
very instructively point out that the decision on where to look for NCT ID in a et al publication is a specificity/sensitivity trade-off.Ironically, the restriction of search to SI, which was originally meant to exclude studies not reporting main results does sometimes include false results (as exemplified by EORTC 55971), and more importantly, the reverse can also be true.Extending the search to the whole abstract, however, might allow even more false results to enter.Interestingly, Powell-Smith is somewhat vague about this issue stating that "in our experience approximately 1.5% of PubMed records include a valid nct_id list in the abstract, but not the Secondary Source ID field" without further details.I am positive that additional research into this topic would be beneficial.

Coens
are quoting Powell-Smith to justify that EORTC was acting correctly when NCT et al et al IDs were published in the text, regardless of where it appears ("The authors state that since 2005 >>all major medical journals (through the International Committee of Medical Journal Editors) have required trials to be registered, and all trials should include their registry ID in the text.<<The majority of our trials, as identified by the tracker, fulfill these criteria".)This quotation is somewhat misleading to suggest that the ID can be anywhere in the text (and not necessarily in SI -as is the case for many EORTC publication) and it is completely correct this way: MEDLINE's guideline explicitly requires NCT ID to be recorded in that particular field, i.e. the secondary source ID field (as also cited by Powell-Smith ).Thus, TrialsTracker's requirement is not arbitrary, as one et al might believe based on the description of Coens : "several were incorrectly classified due to et al absence of a specific PubMed field provided by the medical journals" -SI is not just "a specific PubMed field".More importantly, in some cases, NCT ID is given not only outside the SI field, but outside the abstract.(E.g.NCT00003941.An even more problematic example is NCT00021450, where the NCT IDs location is behind the paywall.)We can argue whether the script should look for the abstract or only the SI field, but obviously there is no realistic way to scan the full text of the articles, so these cases are clearly invisible to any automated algorithm, no matter how elaborate.
Finally, let me note that the authors -quite rightly -summarize the drawbacks of the automation, but to be balanced, its strengths should have been mentioned: first, that the automated nature allows investigation on a scale in quantity that cannot be achieved -or only through extreme measures -with manual checking, and second, perhaps even more importantly, that the automated algorithm is totally transparent and surely free of any subjective decisions.Finally, even if the algorithm is imperfect, at least it is uniformly imperfect, thus the results are likely comparable in spite of this.As far as the distribution of these false results is similar among sponsors, their TrialsTracker score can still be compared.(That's the reason why one cannot simply correct EORTC's results, for example based on this paper, because that would mean that even this comparability is lost.)I have read this submission.I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com Tracker and found similar results to Competing Interests: Coen et al, namely that the Trials Tracker seriously underestimates publication rates.See http://www.statsguy.co.uk/the-trials-tracker-and-post-truth-politics/ /www.statsguy.co.uk/the-trials-tracker-and-post-truth-politics/I have read this submission.I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.08 February 2017 Reviewer Report https://doi.org/10.5256/f1000research.11319.r19624© 2017 Ferenci T. This is an open access peer review report distributed under the terms of the Creative Commons , which permits unrestricted use, distribution, and reproduction in any medium, provided the original Attribution Licence work is properly cited.Tamas Ferenci John von Neumann Faculty of Informatics, Óbuda University, Budapest, Hungary This paper from Coens discusses an important aspect of the recently published TrialsTracker et al database/analysis of the EBM Data Lab (University of Oxford).The paper introducing it from Drs.

References 1 .
Powell-Smith A, Goldacre B: The TrialsTracker: Automated ongoing monitoring of failure to share clinical trial results by all major companies and research institutions. .2016; : 2629 F1000Res 5 PubMed | Abstract Publisher Full Text 2. ABPI response to Trials Tracker data on clinical trials.Association of the British Pharmaceutical

4 5
Page 13 of 14 F1000Research 2017, 6:71 Last updated: 17 MAY 2019 2. ABPI response to Trials Tracker data on clinical trials.Association of the British Pharmaceutical .Industry (ABPI) Reference Source 3. Tweet from Sanofi Pasteur.Reference Source 4. Users' guide to detecting misleading claims in clinical research reports. .2004; (7474): 1093-6 BMJ 329 | PubMed Abstract Publisher Full Text 5. Clinical Trial Registry Numbers in MEDLINE®/PubMed® Records. .US National Library of Medicine Reference Source No competing interests were disclosed.Competing Interests: