F1000Research F1000Research 2046-1402 F1000 Research Limited London, UK 10.12688/f1000research.6312.2 Research Note Articles Methods for Diagnostic & Therapeutic Studies Statistical Methodologies & Health Informatics Funding source and primary outcome changes in clinical trials registered on ClinicalTrials.gov are associated with the reporting of a statistically significant primary outcome: a cross-sectional study

[version 2; peer review: 2 approved]

 Ramagopalan Sreeram V https://orcid.org/0000-0002-4766-5160 a 1 Skingsley Andrew P. 2 Handunnetthi Lahiru 3 Magnus Daniel 4 Klingel Michelle 5 Pakpoor Julia 3 Goldacre Ben b 5 SVR Consulting, London, UK Imperial College Medical School, London, UK Medical Research Council Functional Genomics Unit and Department of Physiology, Anatomy and Genetics, University of Oxford, Oxford, UK School of Social and Community Medicine, University of Bristol, Bristol, UK Department of Epidemiology, London School of Hygiene and Tropical Medicine, London, UK sreeram@ramagopalan.net ben@goldacre.net

SVR and BG conceived and designed the study. SVR, JP, LH, APS, MK and DM analysed the data. SVR and BG interpreted the data. SVR drafted the article. All authors revised the article and gave final approval for publication.

Competing interests: No competing interests were disclosed.

 24 4 2015 2015 4 80 24 4 2015 Copyright: © 2015 Ramagopalan SV et al. 2015 This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Prevalence of primary outcome changes in clinical trials registered on ClinicalTrials.gov: a cross-sectional study 10.12688/f1000research.3784.1

Background: We and others have shown a significant proportion of interventional trials registered on ClinicalTrials.gov have their primary outcomes altered after the listed study start and completion dates. The objectives of this study were to investigate whether changes made to primary outcomes are associated with the likelihood of reporting a statistically significant primary outcome on ClinicalTrials.gov.

Methods: A cross-sectional analysis of all interventional clinical trials registered on ClinicalTrials.gov as of 20 November 2014 was performed. The main outcome was any change made to the initially listed primary outcome and the time of the change in relation to the trial start and end date.

Findings: 13,238 completed interventional trials were registered with ClinicalTrials.gov that also had study results posted on the website. 2555 (19.3%) had one or more statistically significant primary outcomes. Statistical analysis showed that registration year, funding source and primary outcome change after trial completion were associated with reporting a statistically significant primary outcome.

Conclusions: Funding source and primary outcome change after trial completion are associated with a statistically significant primary outcome report on ClinicalTrials.gov.

 clinical trials funding primary outcome The author(s) declared that no grants were involved in supporting this work. Amendments from Version 1

We thank the reviewers for their comments. We have updated the manuscript with the following: 1) Defining how a primary outcome change was defined. 2) Highlighting the differences in effect direction for associated variables. 3) Providing an analysis controlling for study phase (results materially unchanged) 4) Performing the analysis after excluding potential non-inferiority trials (results materially unchanged)

 Introduction

Clinical trials provide the principal method with which to assess the effectiveness of therapeutic strategies 1 . An important principle in the good conduct of clinical trials is that a summary of the trial protocol, with a pre-defined primary outcome, should be freely available before the study commences 1 . In February 2000, the United States (US) Food and Drug Administration (FDA) created an online clinical trials registry named ClinicalTrials.gov 2 . We and others have shown a significant proportion of interventional trials registered on ClinicalTrials.gov have their primary outcomes altered after the listed study start and completion dates 3, 4 . In this extended analysis, we sought to investigate whether changes made to primary outcomes are associated with the likelihood of reporting a statistically significant primary outcome on ClinicalTrials.gov.

 Methods

We used R ( http://cran.r-project.org/web/packages/rclinicaltrials/vignettes/basics.html) to download data from all completed interventional clinical studies registered with ClinicalTrials.gov as of 20th November 2014, as previously described 3 . New to this study, we also downloaded data concerning study results for these trials; specifically the ‘p value’ fields from the ‘study results’ tab for primary outcomes.

We searched for potential non-inferiority studies by text searching for the word inferiority in the title.

Changes in primary outcomes were defined as previously described 3 . A study was classified as not having a primary outcome changed if the original primary outcome was listed as ‘same as current’. Probable funding source was derived using the algorithm previously described 3 .

A trial having a statistically significant primary outcome was defined as a trial having a P value less than 0.05 in the p value field in the study results tab for any primary outcome.

We used logistic regression to calculate odds ratios (ORs) and 95% confidence intervals (95% CI) for comparisons between significant primary outcome and non-significant primary outcome groups, using registration date, primary outcome change after study completion and funding source as explanatory variables. P-values <0.05 were interpreted as significant. Statistical analyses were conducted using the STATA 12.0.0 software.

 Results

All clinical studies classified as ‘interventional studies’ registered with ClinicalTrials.gov as of 20 th November 2014 are shown. Probable funding source was derived using the algorithm previously described 3 . A statistically significant primary outcome was defined as a trial having a P value less than 0.05. 1=yes; 0=no; blank=no info in all columns except “studyphase” and “sponsortype (0=public; 1=industry; 2=mixed). pom: primary outcome measure; sig: significance.

As of 20 November 2014, 13,238 completed interventional trials were registered with ClinicalTrials.gov that also had study results posted on the website. The trials were registered between 1999 and 2014 and 2555 (19.3%) had one or more statistically significant primary outcomes. There were 3934 (29.7%) trials classed as non-industry funded, 1569 (11.9%) as mixed and 7735 (58.4%) as industry funded. 12632 (95.4%) trials had a change in the primary outcome reported at initial registration; 12243 (92.5%) of these occurred after the trial completion date.

Statistical analysis showed that registration year, funding source and primary outcome change after trial completion were associated with the reporting a statistically significant primary outcome ( Table 1). Mixed funding and increased year of registration (i.e. more recent calendar time) were associated with a decreased odds of reporting a statistically significant primary outcome. A primary outcome change and industry funding were associated with an increased odds of reporting a statistically significant primary outcome. We identified 123 trials that had inferiority in the title. Removing these studies from the analysis did not materially change the results.

When including study phase in the analysis (10633 studies with study phase data available), mixed funding and registration year became non-significant.

Association of funding status and primary outcome change after trial completion with reporting a statistically significant primary outcome.
Odds Ratio (95% confidence interval) P value
Public funding 1
Mixed funding 0.79 (0.67–0.94) 0.008
Industry funding 1.39 (1.25–1.54) <0.001
No primary outcome change 1
Primary outcome change after completion date 1.53 (1.12–2.10) 0.008
Registration year (per additional year) 0.97 (0.95–0.99) 0.006
Conclusions

We found that the reporting of statistically significant outcomes on ClinicalTrials.gov was more likely for trials with primary outcomes that had been changed and also those funded by industry. Previous studies have documented these associations 5, 6 , and we confirm these using ClinicalTrials.gov data. There are limitations to our analyses- we have not investigated in any detail the nature of the primary outcome change and the potential effect this would have on the statistical analysis/outcomes. As discussed previously 3 , some primary outcome changes that we have identified may be typographical/semantic and may not reflect actual changes to the nature of the outcome. We also did not look specifically to see whether a changed primary outcome was the one with a statistically significant finding, just whether a statistically significant finding was found for any primary outcome for the study. The vast majority of studies with results reported on ClinicalTrials.gov had a primary outcome change. This suggests that these trials are ones where the registrations have more diligent data updating. Nevertheless, this should be seen in equal measure for trials with and without statistically significant primary outcomes. In summary, funding source and primary outcome changes are associated with the reporting of statistically significant primary outcomes on ClinicalTrials.gov.

 Data availability

The data referenced by this article are under copyright with the following copyright statement: Copyright: � 2015 Ramagopalan SV et al.

Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

F1000Research: Dataset 1. Dataset of funding source, primary outcome changes and statistical significance of clinical trials registered on ClinicalTrials.org, 10.5256/f1000research.6312.d45056 7

Pocock SJ : Clinical trials: a practical approach. Wiley;1983;286. Reference Source Zarin DA Tse T Williams RJ : The ClinicalTrials.gov results database -- update and key issues. N Engl J Med. 2011;364(9):852–60. 21366476 10.1056/NEJMsa1012065 3066456 Ramagopalan S Skingsley AP Handunnetthi L : Prevalence of primary outcome changes in clinical trials registered on ClinicalTrials.gov: a cross-sectional study [v1; ref status: indexed, http://f1000r.es/34l]. F1000Res. 2014;3:77. 25075294 10.12688/f1000research.3784.1 4032105 Huić M Marušić M Marušić A : Completeness and changes in registered data and reporting bias of randomized controlled trials in ICMJE journals after trial registration policy. PLoS One. 2011;6(9):e25258. 21957485 10.1371/journal.pone.0025258 3177887 Bhandari M Busse JW Jackowski D : Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. CMAJ. 2004;170(4):477–80. 14970094 332713 Mathieu S Boutron I Moher D : Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009;302(9):977–84. 19724045 10.1001/jama.2009.1242 Ramagopalan SV Skingsley AP Handunnetthi L : Dataset 1 in: Funding source and primary outcome changes in clinical trials registered on ClinicalTrials.gov are associated with the reporting of a statistically significant primary outcome: a cross-sectional study. F1000Research. 2015. Data Source 10.5256/f1000research.6895.r8561 Reviewer response for version 2 Korenstein Deborah 1 Referee Division of General Medicine, Mount Sinai School of Medicine, New York, NY, USA

Competing interests: No competing interests were disclosed.

 5 5 2015 Copyright: © 2015 Korenstein D 2015 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve

The authors have adequately addressed my concerns.

Reviewer Expertise:

NA

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

 10.5256/f1000research.6895.r8472 Reviewer response for version 2 Wale Janet 1 Referee Cochrane Collaboration Consumer Network, Brunswick, VIC, Australia

Competing interests: No competing interests were disclosed.

 27 4 2015 Copyright: © 2015 Wale J 2015 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve

The authors have clarified a number of the issues raised.

Reviewer Expertise:

NA

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

 10.5256/f1000research.6769.r8116 Reviewer response for version 1 Wale Janet 1 Referee Cochrane Collaboration Consumer Network, Brunswick, VIC, Australia

Competing interests: No competing interests were disclosed.

 2 4 2015 Copyright: © 2015 Wale J 2015 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

This Research Note is an extension of the authors' 2014 article (reference 3). It has a very clearly defined question, whether changes made to the primary outcomes are associated with statistically significant primary outcomes. The present data therefore includes only completed interventional studies on clinicaltrials.gov.

The second paragraph of the Methods section refers to the 2014 article. This is unhelpful, particularly as it is not clear from the 2014 article how 'changes in primary outcomes' are defined. The final paragraph of the Results section states that registration year, funding source and primary outcome change after trial completion were associated with a significant primary outcome - yet these are in opposite directions; and registration year is complex (looking at the data and 2014 article). That is brevity has taken over from clarity.

Some of the limitations are included in the Conclusions: what exactly the changes were ('semantics' versus actual change; whether the changed outcome was the statistically significant outcome reported). The authors have not gone on to analyse their results by phase of trial; if the trials are randomised controlled trials, or otherwise. Another important question is how many of the completed trials have reported their results within a set timeframe (one year/two years), that is what about the trials that have not reported their results?

Has the number of industry funded trials increased over time compared with mixed and public funded trials? In plain language, what is the extent of the problem?

Reviewer Expertise:

NA

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

 10.5256/f1000research.6769.r8112 Reviewer response for version 1 Korenstein Deborah 1 Referee Division of General Medicine, Mount Sinai School of Medicine, New York, NY, USA

Competing interests: No competing interests were disclosed.

 30 3 2015 Copyright: © 2015 Korenstein D 2015 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

Ramagopalan and colleagues have expanded on their previous work to assess the relationship between changes in the primary endpoint on clinicaltrials.gov and both funding source and a “positive” trial result. The authors found that changes to the primary listed endpoint were associated with both industry funding and with a positive outcome.

Their sample included completed interventional clinical trials listed on clinicaltrials.gov. They defined trials as having a positive result if they had a listed p-value <0.05. This may be problematic since it appears that their sample included non-inferiority trials (though it is not clear how many) and for these trials a non-significant p-value may indicate a “positive” (or at least non-inferior) result. Since there have been growing numbers of non-inferiority trials published in recent years, this may be a substantial issue. The authors may want to consider identifying non-inferiority trials and considering their results differently, or at least reporting the prevalence of non-inferiority trials if possible.

Aside from this methodologic weakness the other methods are rather straightforward and clear. However, the authors found that 95.4% of trials had changed the primary outcome at some point during the registration period. In contrast, in their previous work the same authors found that 32% of trials registered with clinicaltrials.gov had changed the primary endpoint. The reason for this dramatic difference is not clear, and the authors do acknowledge that the vast majority of studies changed their primary endpoint and that many of the changes may have been trivial. Further, in spite of this surprising finding the authors still found significant associations. However, the near-total prevalence of changes to the primary endpoint certainly suggests that changing a primary endpoint in the registry is highly routine and likely does not reflect fundamental change to the study. This weakens the relevance of the findings.

Reviewer Expertise:

NA

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.