A call to arms to help heal medicine’s greatest ailment - Publication bias and inadequate research transparency

Martin Mayer

doi:10.12688/f1000research.7037.1

Home Browse A call to arms to help heal medicine’s greatest ailment - Publication...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Opinion Article

A call to arms to help heal medicine’s greatest ailment - Publication bias and inadequate research transparency

[version 1; peer review: 1 approved with reservations]

Martin Mayer

PUBLISHED 16 Sep 2015

Author details Author details

Department of Physician Assistant Studies, College of Allied Health Sciences, East Carolina University, Greenville, NC, 27858-4353, USA

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the All trials matter collection.

Abstract

The paradigm of evidence-based medicine has made impressive advancements since its conception and implementation, but publication bias and issues with inadequate research transparency have remained persistent and pestilent problems. These closely-related issues have markedly detrimental effects on the evidence base from which researchers operate and medical providers make health care decisions, and this can result in involuntary violation of professional and ethical duties and supererogatory motives to serve the public; likewise, it puts patients at risk of receiving medical interventions or advice based on incomplete or ill-understood evidence. By informing readers about the scope of these issues, the failed attempts to correct these issues, and current efforts underway (including a measure in which the lay population can participate), this article serves as a call to arms to help eradicate these incredibly important problems.

Keywords

publication bias, research transparency, evidence-based medicine, medical ethics, research ethics, research misconduct

Corresponding author: Martin Mayer

Competing interests: Truly none, but in the interest of full disclosure, I am a member of the U.S. Board of BMJ Fellows. I do not receive any compensation from The BMJ or anyone else as a result of this. I disclose this here since this article cites works published in The BMJ and ultimately draws attention to their efforts in this arena.

Grant information: The author(s) declared that no grants were involved in supporting this work.

Copyright: © 2015 Mayer M. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Mayer M. A call to arms to help heal medicine’s greatest ailment - Publication bias and inadequate research transparency [version 1; peer review: 1 approved with reservations]. F1000Research 2015, 4:825 (https://doi.org/10.12688/f1000research.7037.1) First published: 16 Sep 2015, 4:825 (https://doi.org/10.12688/f1000research.7037.1) Latest published: 16 Sep 2015, 4:825 (https://doi.org/10.12688/f1000research.7037.1)

Introduction

The problem of a certain proclivity to emphasize “positive” or “successful” findings was specifically mentioned at least as early as 1909 in The Boston Medical and Surgical Journal (Figure 1).^1,2 There is even discussion of the importance objectively recording both positive and negative results as early as 1792,³ and even Diagoras of Melos of 500 BC/BCE recognized that recording only positive outcomes can be misleading.⁴

Figure 1. An editorial published in 1909 in The Boston Medical and Surgical Journal.^1,2

Interested readers can access the full article through the open-access James Lind Library (http://www.jameslindlibrary.org/editorial-1909/). The excerpts are reproduced here under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Sterling appears to have been one of the first to attempt to assess publication bias in a more systematic way in his seminal 1959 article.⁵ Of 362 research reports in four psychology journals, 294 used tests of statistical significance. Impressively, 97.3% of these (286/294) were “positive” (i.e. rejected the null hypothesis); thus, only 2.7% (8/294) were “negative” (i.e. failed to reject the null hypothesis). He astutely noted:

“Some onus appears to be attached to reporting negative results. Certainly such results occur with lesser frequency in the literature than they may reasonably be expected to happen in the laboratory – even if it is assumed that all experimenters are outstandingly clever in selecting hypotheses.”^5(p34)

Sterling also noted this issue permeated other disciplines. Unfortunately, when Sterling looked again at the close of his career, he found essentially no difference.⁶

Despite these early warnings, publication bias has continued unrestrained. Systematic reviews demonstrate publication bias is indisputably pervasive in the medical literature, with the two most recent systematic reviews from 2010 and 2013 finding around 50% or more of studies go unpublished.^7–9 Evidence of inadequate research transparency is similarly worrisome,^7,10–12 and examination of specific examples of these two closely-related issues in action adds poignant tangibility to their seriousness.¹³ These issues distort the evidence base from which medical providers and researchers should be operating, and therefore put patients at risk of direct or indirect harm. Unfortunately, previous efforts to rectify these issues have failed.

Well-intended but ineffective remedial attempts

In 2004, the International Committee of Medical Journal Editors (ICMJE) proclaimed a clinical trial must register “at or before the onset of patient enrollment” in order to be considered for publication in its member journals.^14(p1250) The ICMJE urged nonmember journals to adopt the same policy. At the time, ClinicalTrials.gov was the only suitable registry.¹⁴ The ICMJE penned another editorial a year later outlining what satisfactory registration entailed.¹⁵ Unfortunately, this has not worked – even for member journals – because the decree has not been upheld.^16–19 The most recent appraisal of this issue looked at five psychiatry journals that have the highest impact factors in the field of psychiatry and also adhere to ICMJE guidelines: The American Journal of Psychiatry, Archives of General Psychiatry/JAMA Psychiatry, Biological Psychiatry, Journal of the American Academy of Child and Adolescent Psychiatry, and The Journal of Clinical Psychiatry.¹⁹ Using a search period of January 1, 2009, through July 31, 2013, researchers identified a total of 181 published clinical trials that required prospective registration as aforementioned. Only 60 of these 181 clinical trials (33.1%) actually prospectively registered the trial with the primary outcome(s) clearly defined; only 26 of these 181 trials (14.4%) prospectively registered with the primary outcome(s) clearly defined, had no discrepancies between the protocol and the article, and did not retrospectively modify the primary outcome(s) in the registry.

Title VIII, Section 801 of the Food and Drug Administration Amendments Act of 2007 (FDAAA of 2007) states all applicable clinical trials must register prior to initiation (technically within 21 days of the first patient being enrolled).^20,21 The FDAAA of 2007 also requires trial results to be posted in summary form to ClinicalTrials.gov within 12 months of trial completion unless the trial meets certain exemption criteria and the responsible party applies for and is granted an extension.^20,21 Applicable clinical trials include clinical trials of drugs or biologic agents subject to FDA regulation (other than phase I trials), clinical trials of devices subject to FDA regulation (other than small feasibility studies), and postmarketing surveillance studies of pediatric devices required by the FDA.^20,21 Generally, applicable trials also meet one of the following criteria: one or more trial sites are in the U.S.; the trial is being completed as part of an investigational new drug application or investigational device exemption; or the trial involves a drug, biologic agent, or device that is manufactured in the U.S. or its territories.^20,21 The FDAAA of 2007 only applies to trials starting after September 27, 2007, or considered ongoing as of December 26, 2007.^20,21 These disappointingly insufficient date boundaries already set up the FDAAA of 2007 for a partial success at best, as they do not account for the vast amount of clinical trial data that accumulated before the mandate’s effective start date.

Regarding extensions, permission may be granted to delay submission up to 30 days after product approval if the product has not yet been approved by the time of trial completion (termed a “certification of initial use”). Similarly, a “certification of new use” may be submitted if there are plans to seek FDA approval, clearance, or licensure for new use of an already-approved medical product; in such instances, submission may be delayed for either up to two years or 30 days after one of the following occurs: the FDA determines approval status, the FDA issues a response letter, or the application is withdrawn.^20,21

The penalty for failing to post results as mandated is “not more than” $10,000 initially, which is followed by a 30-day period to correct the infraction, and then “not more than” $10,000 per day thereafter while in violation of the edict.^20(p98)

Unfortunately, these measures have also failed to provide an adequate remedy.^22–28 Importantly, the failure to properly publish studies in general, the failure to post results to ClinicalTrials.gov even when mandated by the FDAAA of 2007, and the failure to publish even when registered with ClinicalTrials.gov are not failures unique to industry; indeed, there is evidence that studies funded by academic and government organizations also suffer considerably from publication bias, and in some cases, such studies may be worse in this regard than industry-funded studies.^22–28 In spite of this, the aforementioned fines have never been imposed.

The most recent appraisal of this issue was published by Anderson and colleagues, and although it certainly adds to our knowledge about ClinicalTrials.gov, it ultimately paints the same unfortunate picture: ClinicalTrials.gov has failed.²⁸

Anderson and colleagues identified 32,656 trials that were very likely to fall under the mandated reporting requirements of the FDAAA of 2007; the authors termed these trials highly likely applicable clinical trials (HLACTs). Such verbiage was necessary because they could not be absolutely certain which trials were subject to the FDAAA of 2007 based on the publicly-available information. Such an impediment carries an inherent and saddening irony given the inadequacies of ClinicalTrials.gov thereby suggested. In order to maximize accuracy given this obstacle, they used an algorithm based on input from the National Library of Medicine via personal communication with Deborah Zarin, the Director of ClinicalTrials.gov.

After whittling this sample via exclusion criteria (trial status, trial completion date, or data completeness in registry entry), their final sample was 13,327 HLACTs completed or terminated between January 1, 2008, and August 31, 2012. The final follow-up time for the five-year study period was September 27, 2013. Industry funded 65.6% of the HLACTs, the National Institutes of Health (NIH) funded 14.2%, and other government or academic institutions funded 20.2%.

Only 1,790 trials (13.4%) reported results within 12 months, and even when expanding the consideration to the five-year study period, only 5,110 (38.3%) reported results at any time up to September 27, 2013.

At 12 months after trial completion, only 818 trials (6.1%) had a legally-acceptable delay; furthermore, by the end of September 27, 2013, only 2,100 trials (15.8%) had requested a delay or certified their qualification for a delay in reporting to ClinicalTrials.gov.

These numbers, unacceptable as they may be, are still better than the corresponding numbers for the 25,646 non-HLACTs, where only 1,287 (5.0%) and 2,473 (9.6%) reported results within 12 months and at any time during the five-year period, respectively.

Their analysis also provides additional evidence that industry is not the only guilty party. In fact, in their analysis, industry was best at reporting, albeit still with a wholly unacceptable performance (see Table 1 for reporting rates based on trial funding).

In addition to the ICMJE proclamation and the FDAAA of 2007 proving to be, thus far, inadequate remedies, a recent survey suggested reviewers may also fall short.²⁹ The usable survey response rate was 37.5% (1,136 respondents with usable surveys out of 3,033 potential participants completing the survey), and 676 respondents indicated they had reviewed a clinical trial in the past two years; unfortunately, only 232 out of the 676 (34.3%) reported assessing trial registry information when reviewing a manuscript.²⁹

Table 1. Compliance estimates from Anderson and colleagues’ analysis of compliance with reporting results for highly likely applicable clinical trials based on funding and time period.²⁸

	Total (n = 13,327)	Industry-funded (n = 8,736)	NIH-funded (n = 1,899)	Other government or academic institution-funded (n = 2,692)
Results reported within 12 months of completion	1,790 (13.4)	1,483 (17.0)	153 (8.1)	154 (5.7)
Results reported within five-year study period	5,110 (38.3)	3,624 (41.5)	739 (38.9)	747 (27.7)

Data presented are number (percentage) in each group and come from Table 2 of Anderson and colleagues’ analysis.²⁸

In late 2014, the U.S. Department of Health and Human Services (HHS) and the NIH proposed changes that seek to: clarify certain aspects of the current legislation; increase somewhat the elements of applicable trials that must be reported; and expand the definition of trials that are required to register and provide results, including a new NIH policy requiring all clinical trials receiving NIH funding to register and submit results in a manner similar to that required of trials falling under the FDAAA of 2007 mandate.^30–32 While this expansion seems at first glance to be desirable and optimistic, one cannot help but notice the utter failure to enforce the FDAAA of 2007 since its inception; thus, it is difficult to have faith that an expansion of responsibilities will help in any material way unless there are concurrent and enforced measures to ensure adherence, and the current state of affairs – even with the recently-proposed changes – leaves much to be desired, as enforcement remains an unaddressed issue. Furthermore, these changes still do not address trial data prior to the effective start date of the FDAAA of 2007, and beyond date limitations, these changes still only guarantee access to the elements that must be reported, not all the data.

Recently, the European Medicines Agency adopted a policy change requiring publication of full clinical study reports (CSRs) for all applications resulting in a new drug approval after January 1, 2015.^33,34 While certainly a big step forward, it also does not account for any trial data prior to its date of enactment.

This should not be seen as a simple pessimistic accusation of an entirely lackadaisical approach on the part of those intimately involved in the above measures; however, it is an unwavering and unequivocal assertion that these measures, in their current state and execution, are ineffective or incomplete remedies. However, we must persistently pursue a remedy for these issues until true resolution occurs, and there are a number of current measures that deserve consideration.

Potential remedies currently being tested

During its pursuit of the clinical trial data on oseltamivir (Tamiflu®), The BMJ launched the Open Data campaign in commitment to clinical research transparency (http://www.bmj.com/open-data), and Peter Doshi (an associate editor at The BMJ) leads the Restoring Invisible and Abandoned Trials (RIAT) initiative.³⁵ The RIAT initiative is a call to publish or be published; specifically, it calls for the original researchers involved in known-to-be abandoned or misreported trials to correctly publish (or republish, if necessary) such trials in the peer-reviewed literature. In the absence of such corrective behavior, the RIAT initiative describes the mechanism by which it will pursue publication independent of the original researchers. The first trial utilizing the RIAT initiative was published in May of 2014,³⁶ and one hopes the RIAT initiative will remain perpetually active.

Formally launched in 2013, the AllTrials initiative has gained much momentum and has played a remarkable role in combating these issues (http://www.alltrials.net/). AllTrials is a joint initiative of (in alphabetic order): Bad Science, The BMJ, the Centre for Evidence-based Medicine, the Cochrane Collaboration, the James Lind Initiative, PLoS, and Sense About Science. It is being led in the U.S. by Dartmouth’s Geisel School of Medicine and the Dartmouth Institute for Health Policy and Clinical Practice, and its official U.S. launch occurred just this year in late July. The AllTrials initiative calls for: (1) registration of all trials, including a summary of the protocol, before the trial begins (with past trials being registered retrospectively); (2) summaries of trial results being publicly available within one year of completing the trial, with the results being posted where the trial was registered (with results of past trials being made available now); and (3) full reports (e.g. CSRs) being made publicly available whenever they exist or are created (http://www.alltrials.net/find-out-more/all-trials/). The AllTrials initiative also discusses measures of monitoring and enforcement, including how current infrastructures could be better utilized. The AllTrials initiative does not call for releasing individual patient data, and it respects the potential need for redaction of identifying information (such redaction, where necessary, is not an unduly arduous task). As of the time of this writing, the AllTrials initiative has 86,030 individual supporters and 612 organizations (http://www.alltrials.net/supporters/). The AllTrials initiative remains very active in this arena, and its petition remains open for anyone to sign, even those outside the medical and scientific community.

Among the organizations to sign the petition is GlaxoSmithKline (GSK), which was a surprising and welcome addition. True transparency would be a formidable and promising step in the right direction, and Patrick Vallance, President of Pharmaceuticals Research and Development at GSK, and Sir Andrew Witty, Chief Executive Officer of GSK, have both committed to greater transparency; however, the full extent of GSK’s commitment has yet to be fully elucidated.^37–40 For instance, a recent perspective piece paints a very positive picture of the raw data access mechanism for GSK-sponsored trials, a mechanism that is now also being used by several other pharmaceutical companies.³⁸ A purportedly independent panel oversees this process, including being responsible for reviewing and approving applications for access; however, all the authors of this perspective piece (who are also members of the panel) have industry ties declared in the associated disclosures form.³⁸ This is not simply a captious attempt to “poison the well,” nor does it imply the panel members are inherently nefarious; however, it remains an inarguably important consideration when reading the perspective piece and considering the independence of the process. The novelty of this mechanism may result in an ongoing evolution of the process, and Zarin has raised important questions and concerns.³⁹ Likewise, the only perspectives available from individuals who have been on the other side of this process are those of Jon Jureidini and colleagues, and they suggest an unsatisfactory experience that raised uncertainties.^37,40 (Jureidini and colleagues took on the RIAT project of rewriting GSK’s Study 329 on paroxetine in adolescents, which is now in press [written communication with Jon Jureidini on August 25, 2015, with permission granted to publish this information].)

Early in January of 2015, a committee from the Institute of Medicine released a report on clinical trial data sharing.⁴¹ Though the committee admits its extensive writing on the matter is not necessarily an all-encompassing solution, it nevertheless provides meaningful discussion concerning basic principles, operational guidance, and recommendations to help – as the title of the report says – maximize benefit and minimize risk of clinical trial data sharing. Four times in the report, they call for fostering “a culture in which data sharing is the expected norm.”^{41(p2,5,23,65)}

We need that culture.

This was echoed when the World Health Organization (WHO) updated its position on public disclosure of clinical trial results.⁴² It reiterates the need for prospective and transparent registration and the ethical imperative of reporting results, and it also calls for: (1) submitting results for publication in a peer-reviewed journal via an open-access mechanism (unless there is a specific reason why an open-access mechanism cannot be used) within 12 months of study completion, with publication expected within 12 months after submission; (2) making results publicly available via another mechanism within 24 months of trial completion if (1) is not possible for some reason; (3) publishing key elements in an open-access clinical trial registry within 12 months of study completion; and importantly, (4) reporting as-of-yet unpublished trials in an open-access clinical trial registry at minimum, with urging for concomitant publication in a peer-reviewed journal.

Conclusion

Issues with publication bias and tainted trial transparency loom large and threaten the very core of evidence-based practice. Iain Chalmers – just recipient of The BMJ’s lifetime achievement award in 2014⁴³ and someone who has devoted an enormous amount of his career to these issues – wrote a seminal article in 1990 claiming that underreporting research is scientific misconduct.⁴⁴ This better-known piece actually followed his 1985 correspondence where he proposed outlawing the term “negative trial” and rightly regarded all well-done trials, regardless of their outcome, as being “positive contributions to knowledge.”^45(p1002) Sadly, his words of cautionary wisdom and the aforementioned cautionary glimpses offered by others have been ignored for far too long by far too many.

In addition to being scientifically odious, these issues are in direct violation of the World Medical Association’s Declaration of Helsinki, which is supposed to serve as the authority on ethics in medical research involving human subjects (Figure 2).⁴⁶ Continuing to tolerate these problems is not only an ethical blemish of the worst kind, but is also: wholly disrespectful to those who participate in clinical trials with the belief their participation will help improve care; a cause of inefficient use of research time and funding; and perhaps most importantly, a threat to providers’ ability to provide their patients with the best evidence-based care possible.

Figure 2. Key principles from the World Medical Association’s Declaration of Helsinki addressing the need for transparency and proper dissemination of research involving human subjects.

Reproduction of directly-applicable components was approved by the World Medical Association, but the World Medical Association implores readers to read and follow the Declaration of Helsinki in full (http://www.wma.net/en/30publications/10policies/b3/).⁴⁶

Previous efforts to address these issues have fallen short. Current movements to combat these issues have gained vigor, and one hopes they will not ultimately fall short like their predecessors. The fight for transparency has had many noteworthy contributors, as recently summarized in a tribute feature.³⁴ The contributions of Iain Chalmers and Peter Doshi have been briefly outlined, but other noteworthy contributors (in alphabetic order) are Douglas Altman, Kay Dickersin, Fiona Godlee, Ben Goldacre, and Peter Gøtzsche. This list is certainly incomplete, and importantly, all the current contributors and their efforts, while valiant and incredibly important, are still not enough. Without a unified voice from the medical and scientific communities and the general public, even the most concerted of efforts are at risk of being one day catalogued as well-intended but ultimately ineffective remedies.

The movement to end publication bias and inadequate research transparency has gained what some might consider to be unprecedented momentum, but it still remains in a critical condition requiring much additional support. Unless and until we fully eradicate these issues, medicine too remains in a perilous state. Indeed, publication bias and inadequate research transparency represent a gaping wound in the body of evidence from which researchers operate and medical providers make decisions regarding care. We need proper closure and healing of the wound – we owe it to our patients and those who participate in research to end these issues completely; anything less than that will never allow for such healing to occur.

Competing interests

Truly none, but in the interest of full disclosure, I am a member of the U.S. Board of BMJ Fellows. I do not receive any compensation from The BMJ or anyone else as a result of this. I disclose this here since this article cites works published in The BMJ and ultimately draws attention to their efforts in this arena.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Faculty Opinions recommended

References

1. Editorial. The reporting of unsuccessful cases. Boston Med Surg J. 1909; 161: 263–264. Reference Source
2. Dickersin K: Publication bias: recognizing the problem, understanding its origins and scope, and preventing harm. In: Rothstein HR, Sutton AJ and Borenstein M, eds. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. West Sussex, England: John Wiley & Sons; 2005; 11–33. Publisher Full Text
3. Ferriar J: Medical Histories and Reflections. London: Cadell and Davies, 1792; 1. Reference Source
4. Petticrew M: Diagoras of Melos (500 BC): An early analyst of publication bias. Lancet. 1998; 352(9139): 1558. PubMed Abstract | Publisher Full Text
5. Sterling T: Publication decisions and their possible effects on inferences drawn from tests of significance -- Or vice versa. J Am Stat Assoc. 1959; 54(285): 30–34. Publisher Full Text
6. Sterling TD, Rosenbaum WL, Weinkam JJ: Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. Am Stat. 1995; 49(1): 108–112. Publisher Full Text
7. Dwan K, Gamble C, Williamson PR, et al.: Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One. 2013; 8(7): e66844. PubMed Abstract | Publisher Full Text | Free Full Text
8. Song F, Parekh S, Hooper L, et al.: Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010; 14(8): iii, ix–xi, 1–193. PubMed Abstract | Publisher Full Text
9. Hopewell S, Loudon K, Clarke MJ, et al.: Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009; (1): MR000006. Accession # 00075320-100000000-05507. PubMed Abstract | Publisher Full Text
10. Wieseler B, Wolfram N, McGauran N, et al.: Completeness of reporting of patient-relevant clinical trial outcomes: comparison of unpublished clinical study reports with publicly available data. PLoS Med. 2013; 10(10): e1001526. PubMed Abstract | Publisher Full Text | Free Full Text
11. Dwan K, Altman DG, Cresswell L, et al.: Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev. 2011; (1): MR000031. PubMed Abstract | Publisher Full Text
12. Chan AW, Hróbjartsson A, Haahr MT, et al.: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004; 291(20): 2457–2465. PubMed Abstract | Publisher Full Text
13. Mayer M: Peering into the “rabbit hole” of publication bias and inadequate research transparency: Adding tangibility to the abstract [v1; ref status: awaiting peer review, http://f1000r.es/5pp]. F1000Res. 2015; 4: 609. Publisher Full Text
14. De Angelis C, Drazen JM, Frizelle FA, et al.: Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004; 351(12): 1250–1251. PubMed Abstract | Publisher Full Text
15. De Angelis CD, Drazen JM, Frizelle FA, et al.: Is this clinical trial fully registered?--A statement from the International Committee of Medical Journal Editors. N Engl J Med. 2005; 352(23): 2436–2438. PubMed Abstract | Publisher Full Text
16. Mathieu S, Boutron I, Moher D, et al.: Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009; 302(9): 977–84. PubMed Abstract | Publisher Full Text
17. Jones CW, Platts-Mills TF: Quality of registration for clinical trials published in emergency medicine journals. Ann Emerg Med. 2012; 60(4): 458–464.e1. PubMed Abstract | Publisher Full Text
18. Killeen S, Sourallous P, Hunter IA, et al.: Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals. Ann Surg. 2014; 259(1): 193–196. PubMed Abstract | Publisher Full Text
19. Scott A, Rucklidge JJ, Mulder RT: Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. PLoS One. 2015; 10(8): e0133718. PubMed Abstract | Publisher Full Text | Free Full Text
20. The Food and Drug Administration. Title VIII, Section 801, Food and Drug Administration Amendments Act of 2007. 2007. [Pages 82 (http://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf#page=82) through 100 (http://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf#page=100) of the Food and Drug Administration Amendments Act of 2007 (http://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf)] Reference Source
21. U.S. National Institutes of Health. FDAAA 801 requirements. 2014. Last reviewed December 2014. Accessed/link verified December 19, 2014. Reference Source
22. Prayle AP, Hurley MN, Smyth AR: Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ. 2012; 344: d7373. PubMed Abstract | Publisher Full Text
23. Huser V, Cimino JJ: Linking ClinicalTrials.gov and PubMed to track results of interventional human clinical trials. PLoS One. 2013; 8(7): e68409. PubMed Abstract | Publisher Full Text | Free Full Text
24. Jones CW, Handler L, Crowell KE, et al.: Non-publication of large randomized clinical trials: cross sectional analysis. BMJ. 2013; 347: f6104. PubMed Abstract | Publisher Full Text | Free Full Text
25. Law MR, Kawasumi Y, Morgan SG: Despite law, fewer than one in eight completed studies of drugs and biologics are reported on time on ClinicalTrials.gov. Health Aff (Millwood). 2011; 30(12): 2338–45. PubMed Abstract | Publisher Full Text
26. Kirillova O: Results and outcome reporting in ClinicalTrials.gov, what makes it happen? PLoS One. 2012; 7(6): e37847. PubMed Abstract | Publisher Full Text | Free Full Text
27. Ross JS, Mulvey GK, Hines EM, et al.: Trial publication after registration in ClinicalTrials.Gov: a cross-sectional analysis. PLoS Med. 2009; 6(9): e1000144. PubMed Abstract | Publisher Full Text | Free Full Text
28. Anderson ML, Chiswell K, Peterson ED, et al.: Compliance with results reporting at ClinicalTrials.gov. N Engl J Med. 2015; 372(11): 1031–1039. PubMed Abstract | Publisher Full Text | Free Full Text
29. Mathieu S, Chan AW, Ravaud P: Use of trial register information during the peer review process. PLoS One. 2013; 8(4): e59910. PubMed Abstract | Publisher Full Text | Free Full Text
30. U.S. National Institutes of Health, Department of Health and Human Services. 42 CFR Part 11 [Docket Number NIH– 2011– 0003], RIN 0925–AA52, Clinical Trials Registration and Results Submission, Action: Notice of Proposed Rulemaking. Federal Register. 2014; 79(225): Accessed link/verified January 1, 2015. Reference Source
31. U.S. National Institutes of Health. HHS and NIH take steps to enhance transparency of clinical trial results. Published November 19, 2014; Reviewed November 21, 2014. Accessed/link verified November 22, 2014. Reference Source
32. U.S. National Institutes of Health. Summary of HHS/NIH proposals to enhance transparency of clinical trial results. Reviewed November 19, 2014; Accessed/link verified November 22, 2014. Reference Source
33. European Medicines Agency. European Medicines Agency policy on publication of clinical data for medicinal products for human use. 2014. Accessed/link verified January 1, 2015. Reference Source
34. Adams B: The pioneers of transparency. BMJ. 2015; 350: g7717. PubMed Abstract | Publisher Full Text
35. Doshi P, Dickersin K, Healy D, et al.: Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ. 2013; 346: f2865. PubMed Abstract | Publisher Full Text | Free Full Text
36. Treasure T, Monson K, Fiorentino F, et al.: The CEA Second-Look Trial: a randomised controlled trial of carcinoembryonic antigen prompted reoperation for recurrent colorectal cancer. BMJ Open. 2014; 4(5): e004385. PubMed Abstract | Publisher Full Text | Free Full Text
37. Doshi P: Putting GlaxoSmithKline to the test over paroxetine. BMJ. 2013; 347: f6754. PubMed Abstract | Publisher Full Text
38. Strom BL, Buyse M, Hughes J, et al.: Data sharing, year 1--access to data from industry-sponsored clinical trials. N Engl J Med. 2014; 371(22): 2052–4. PubMed Abstract | Publisher Full Text
39. Zarin DA: Participant-level data and the new frontier in trial transparency. N Engl J Med. 2013; 369(5): 468–469. PubMed Abstract | Publisher Full Text
40. Jureidini JN, Nardo JM: Inadequacy of remote desktop interface for independent reanalysis of data from drug trials. BMJ. 2014; 349: g4353. PubMed Abstract | Publisher Full Text
41. Lo B, Coetzee T, Demets D, et al.: Committee on Strategies for Responsible Sharing of Clinical Trial Data, Board on Health Sciences Policy, Institute of Medicine. Sharing clinical trial data: Maximizing benefits, minimizing risk. National Academies Press, 2015. ISBN 978-0-309-31629-3. Accessed/link verified January 17, 2015. Reference Source
42. Moorthy VS, Karam G, Vannice KS, et al.: Rationale for WHO's new position calling for prompt reporting and public disclosure of interventional clinical trial results. PLoS Med. 2015; 12(4): e1001819. PubMed Abstract | Publisher Full Text | Free Full Text
43. Hawkes N: Lifetime achievement award 2014: Sir Iain Chalmers. BMJ. 2014; 348: g2921. PubMed Abstract | Publisher Full Text
44. Chalmers I: Underreporting research is scientific misconduct. JAMA. 1990; 263(10): 1405–1408. PubMed Abstract | Publisher Full Text
45. Chalmers I: Proposal to outlaw the term “negative trial”. BMJ (Clin Res Ed). 1985; 290(6473): 1002. Publisher Full Text | Free Full Text
46. World Medical Association. Declaration of Helsinki: ethical principles for medical research involving human subjects. 2013. Accessed/link verified August 6, 2014. Reference Source

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 16 Sep 2015

Author details Author details

Department of Physician Assistant Studies, College of Allied Health Sciences, East Carolina University, Greenville, NC, 27858-4353, USA

Competing interests

Truly none, but in the interest of full disclosure, I am a member of the U.S. Board of BMJ Fellows. I do not receive any compensation from The BMJ or anyone else as a result of this. I disclose this here since this article cites works published in The BMJ and ultimately draws attention to their efforts in this arena.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Article Versions (1)

version 1

Published: 16 Sep 2015, 4:825

https://doi.org/10.12688/f1000research.7037.1

© 2015 Mayer M. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Mayer M. A call to arms to help heal medicine’s greatest ailment - Publication bias and inadequate research transparency [version 1; peer review: 1 approved with reservations]. F1000Research 2015, 4:825 (https://doi.org/10.12688/f1000research.7037.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 16 Sep 2015

Views

Reviewer Report 02 Oct 2015

Erin Aiello Bowles, Group Health Research Institute, Seattle, WA, USA

Approved with Reservations

https://doi.org/10.5256/f1000research.7575.r10354

This paper outlines an important, rampant issue in public health research – publication bias. The author doesn’t present any scientific data so I can’t judge scientific quality. I appreciate the author’s historical presentation of the issue and the data provided ... Continue reading

The article spends much of the paper focusing on clinical trials. Nearly an entire page is devoted to ClinicalTrials.gov. While this is interesting, it is not the only cause of publication bias. I would cut down some of this information and provide information on additional causes. For example, what about biases from journals? When was the last time a top-tier journal published a non-significant result, even if the result was important. I believe this is where much of the bias lies. Perhaps breaking this into 2 sections would help – one section on bias as a result of the author failing to report results properly (or at all) and one section on bias as a result of journals failing to publish articles that aren’t highly statistically significant. The latter is a difficulty that many authors face that needs more attention if policies will ever change, and may even be part of the reason authors fail to properly report results.
I would also encourage the author to expand on publication bias in observational studies. This is mentioned briefly on page 3 and may be an even bigger problem than publication bias in clinical trials.
The author doesn’t mention that some journals (CEBP being one) have specific calls for “null results” to reduce publication bias. Additional calls and journal policies on actively publishing null results may be one solution to this problem.
The author also doesn’t mention the impact that publication bias has on future research. When a study isn’t published, no one knows. No one knows what the results were or whether the trial worked. So the study may be funded again in a different form and repeated only to find the same null result. This is a complete waste of time and our tax payer dollars, particularly for government-funded studies. The NIH should really crack down on the lack of publications from grants and prohibit funding of future similar studies if the initial results are never published.

Competing Interests: No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 13 Oct 2015

Martin Mayer, Department of Physician Assistant Studies, College of Allied Health Sciences, East Carolina University, Greenville, 27858-4353, USA

13 Oct 2015

Author Response

I am glad Ms. Bowles appreciates the gravity of these issues, and I am grateful for her overall positive review of my article. My article was not designed as a ... Continue reading I am glad Ms. Bowles appreciates the gravity of these issues, and I am grateful for her overall positive review of my article. My article was not designed as a primary research article, and therefore there were no primary scientific data to report. Still, although this does mean one cannot judge scientific quality in terms of methodological appraisal for a primary research article (or a meta-analysis, etc.), I would nevertheless posit my article can still be judged as a positive contribution to the biomedical sciences, as one can ultimately think of science in two broad, interrelated senses: Firstly, science is ultimately a way of thinking about and approaching phenomena that involves asking answerable questions and then systematically and impartially pursuing and using the totality of the best-available evidence to reach an informed and reasonable stance, a stance that should remain appropriately malleable as new evidence accumulates and gets incorporated into the current body of knowledge on a subject; secondly, science is a collaborative endeavor to achieve such goals, to collegially seek out such working truths as best we can.

This collaborative aspect speaks to my sincere appreciation of Ms. Bowles’ review and comments regarding my article. Below, I respond to her comments at length (with the comment[s] to which I am responding indicated in the responses).

Ms. Bowles comments that much attention is paid to ClinicalTrials.gov, but that biases from journals may have been overlooked, and that she believes this is where much of the bias lies (comment 1); she is also concerned that the willingness of certain journals to publish “null results” might have been overlooked, and that this could be part of the solution (comment 3). To be sure, however, no such oversights occurred; although I originally decided to omit an in-depth discussion of such matters in my article for the sake of length and for reasons that will become clear as I respond to these comments, I will gladly expound here, and the available evidence does offer some reassurance and illumination, though it does not support Ms. Bowles’ comments.

Although bias in the peer-review process has been demonstrated,[ref-1] the best-available evidence overall suggests the majority of the problem actually rests with researchers/authors failing to publish or otherwise make available the results of their research, not journals/editors preferentially publishing only studies with “positive”/significant findings.[ref-2]^-[ref-14] Indeed, this is discussed in the 2010 systematic review I cite in my article, which pooled the four studies to date that had investigated this matter.[ref-2] Song and colleagues found a pooled odds ratio of acceptance of 1.06 for manuscripts with “positive”/significant findings versus manuscripts with “negative”/non-significant findings (95% confidence interval [CI], 0.80 to 1.39), which they rightly conclude as corroborating the notion that acceptance of a submitted manuscript was not significantly associated with whether the paper had “positive”/significant or “negative”/non-significant findings. The four studies included in the pooled analysis included two studies investigating general medical journals (JAMA in one study,[ref-3] and The BMJ, The Lancet, and Annals of Internal Medicine in the other[ref-4]) and two studies investigating the American volume of the Journal of Bone and Joint Surgery.[ref-5]^,[ref-6] All these studies individually arrived at the same result as well: once submitted to a journal for consideration, manuscripts with “positive”/significant findings are no more likely to be published than manuscripts with “negative”/non-significant findings.

One of the studies[ref-5] – which investigated only manuscripts on hip or knee arthroplasty submitted to the American volume of the Journal of Bone and Joint Surgery – has a potentially suggestive title, and it drew attention to the significant difference in the average quality of “positive”/significant manuscripts versus “negative”/non-significant manuscripts when quality was assessed by a score derived by the authors from standard criteria used to assess the quality of a piece of evidence (termed the Sackett score by the authors). On average, “negative”/non-significant manuscripts had higher Sackett scores than “positive”/significant manuscripts. When considering the mean level of evidence scale officially required by the journal during manuscript preparation and submission, however, there was no significant difference between “positive”/significant manuscripts and “negative”/non-significant manuscripts. There was no difference in acceptance based on study outcome, and intriguingly, there was no difference in acceptance based on study quality as assessed by either the mean Sackett score or the mean level of evidence scale officially used and required by the journal. This finding causes one to ponder, as one would rightly expect higher-quality manuscripts to be published with greater frequency than lower-quality manuscripts. They discuss the finding of no difference in quality between accepted versus rejected manuscripts thusly:

“Potential explanations for this finding include the possibility that the overall quality of submissions to The Journal is so high that quality differences among them are too subtle to be discerned with use of standard evidence-based-medicine scoring approaches or, more likely, that reviewers make their decisions on the basis of other criteria, such as timeliness, economic impact, or perceived relevance, in addition to those recommended in standard evidence-based-medicine texts.”[ref-5]^(p1016)

However, these postulated explanations still leave one with at least some degree of lingering uncertainty regarding what to think about this finding and how to reconcile this finding with the other assessments and discussions of quality in this study. The authors also acknowledge that assessing manuscript quality and any relationship with acceptance versus rejection as being “outside the scope of [the] primary study end points as defined by [their] a priori hypotheses” and that “until or unless this finding is validated by others it should not be accorded great weight.” [ref-5]^(p1016) (Indeed, investigating study quality was not a part of either of the two clearly-stated primary research hypotheses.)

Furthermore, a large weakness of this study is its lack of attempt to control for potentially influential or confounding variables. Considering all of this, one should be cautious about making any strong conclusions about any findings of manuscript quality or relationships pertaining to manuscript quality and publication from this study, and even setting all this aside, one must also remember this study looked at submissions to one specialty journal and included only studies on hip and knee arthroplasty. Even the subsequent study investigating submissions to the American volume of the Journal of Bone and Joint Surgery noted that, due to the other study’s limitations, “it remains unclear whether publication bias exists in the orthopaedic journal editorial review process.”[ref-6]^(p596) The subsequent study thus set its primary research intent as seeking to determine whether manuscripts with “positive”/significant findings were more likely to be published after controlling for various quality measures, level of evidence, and subspecialty field.[ref-6] In this study and the two studies that investigated general medical journals[ref-3]^,[ref-4] – all of which investigated quality measures and utilized multivariate analyses to control for potentially influential or confounding variables – no difference was found in publication rates for submitted manuscripts with “positive”/significant results versus submitted manuscripts with “negative”/non-significant results. These studies also found that manuscripts of higher quality or level of evidence were significantly more likely to be published.[ref-3]^,[ref-4]^,[ref-6]

Recently, a retrospective review of manuscripts from several journals (The BMJ, Annals of the Rheumatic Diseases, British Journal of Ophthalmology, Gut, Heart, Thorax, Diabetologia, and Journal of Hepatology) was published in PLoS ONE,[ref-7] and although retrospective studies are often viewed as having potential weaknesses compared to prospective studies, the retrospective design offers a unique benefit in this case: the elimination of any Hawthorne effect. Such an effect might occur in the aforementioned prospective studies on this issue, since editors and reviewers might, whether consciously or unconsciously, behave somewhat differently if they are aware their actions are being studied. This retrospective study also found no difference in acceptance for publication for submitted manuscripts with “positive”/significant findings compared to submitted manuscripts with “negative”/non-significant findings, this study considered quality indicators as well, and the finding of no difference was present in both the univariate and multivariate analyses.

However, what if merely looking at publication in a binary manner misses a level of nuance that might still have an impact? What if journals/editors are less enthusiastic about publishing “negative”/non-significant results compared to “positive”/significant results, and therefore, whether unconsciously or consciously, do not pursue the steps to publication as diligently, leading to significant delays in publication? Ioannidis found that, among a cohort of 66 completed multicenter trials focused on human immunodeficiency virus, the 45 trials that had been submitted for publication were published slightly more quickly after submission if they had “positive”/significant results (median of 0.8 years versus 1.1 years; P = 0.04); however, the ultimate clinical significance of an approximate 3.6 month delay is unclear.[ref-8] However, in another study investigating a cohort of 133 trials submitted to JAMA, there was no difference in time from submission to publication for “negative”/non-significant results compared to “positive”/significant results (median of 7.6 months versus 7.8 months, respectively; P = 0.44).[ref-9]

Of course, a potential limitation in these studies is the inherent inability to account for manuscripts/studies that are not even written up and submitted to journals. The theoretical question of what these results would look like if non-submitted manuscripts were submitted is certainly an interesting question, even if only for the sake of refining our understanding of this matter. In the absence of such, however, the aforementioned studies remain our best evidence on the matter, and it would be entirely unfair to hold any speculative counterfactual against the journals/editors.

In summary, when considering what is known about publication bias and what this evidence shows, while the evidence may not allow us to confidently exculpate journals/editors entirely, it surely corroborates the notion that authors/researchers are more culpable than journals/editors.

Further insight can be gained from asking the researchers themselves why they failed to write up and submit a study.[ref-2]^,[ref-10]^-[ref-16] Here too, we see the majority of the problem seems to rest not with the journals/editors, but with the authors/researchers, with the most common reasons for not writing up and submitting a study tending to be lack of time or lack of interest. Both reasons are wholly unacceptable. Being rejected by a journal actually tends to be reported by only a minority, but this too is an unacceptable excuse; manuscript rejection is largely to be expected at some point in one’s publishing endeavors (regardless of the manuscript’s findings/content), and allowing this to be a deterrent is a fault of the researchers/authors, not the journals/editors. Due to cognitive biases, authors/researchers may be more likely to assign fault or biased motives to the journal/editors (and thus may also view things in a more emotionally charged manner) when a manuscript with “negative”/non-significant results is rejected compared to when a manuscript with “positive”/significant results is rejected; however, this is merely a speculative suggestion, albeit one that would seem to fit within what we know about human behavior. In any case, one must remember authors/researchers are specifically called to adhere to the World Medical Association’s Declaration of Helsinki in making the results of their research on humans publicly available (see the second sentence of principle 36 in Figure 2 in my article). Furthermore, at least for clinical trials, ClinicalTrials.gov could serve as a repository for study methods and results even if the authors/researchers do not wish to seek traditional publication or get rejected and decide not to pursue publication elsewhere (although my article discusses mandatory submission, ClinicalTrials.gov also accepts voluntary submissions).[ref-17] Importantly, this trumps any excuse pertaining to journal/editorial rejection.

On the note of ClinicalTrials.gov and the length to which I discuss it (comment 1), I do so because ClinicalTrials.gov was ushered in with great fanfare, and yet it has unequivocally failed and was doomed at the outset to be a partial success at best in its original (and current) form. Such assertions are admittedly bold, so the space I devote to dissecting this issue was seen as necessary, and truthfully, the space I devote to this topic is ultimately relatively limited given the evidence on the matter. However, the failure of ClinicalsTrials.gov to rectify the issues of publication bias and inadequate research transparency does not preclude it from being used in the aforementioned manner; moreover, with some important logistical changes and better oversight and enforcement (all of which I address in my article), ClinicalTrials.gov could actually be a remarkable tool in the fight to end publication bias and inadequate research transparency.

Further questioning the role the journals/editors might play, Ms. Bowles also questions the willingness of a top-tier journal to publish non-significant results, asking when a top-tier journal last published a non-significant result (comment 1, the response to which will also further address comment 3). However, there are several recent examples of studies with “null,” “negative,” or “non-significant” results even within the so-called “Big 5,” including (in alphabetic order) Annals of Internal Medicine,[ref-18] The BMJ,[ref-19]^,[ref-20]Journal of the American Medical Association,[ref-21]^,[ref-22] The Lancet,[ref-23]^,[ref-24] and New England Journal of Medicine.[ref-25]^,[ref-26]

Again, considering all of this, although the evidence cannot entirely rule out journal/editorial bias, the totality of the best-available evidence nevertheless suggests the majority of the problem of publication bias rests more with authors/researchers, not with journals/editors. However, as I discuss clearly in my article, journals/editors are not blameless; indeed, a large area of failure has been the disappointing failure to uphold the ICMJE proclamation.

Ms. Bowles also comments on the seeming focus on clinical trials (comment 1), expresses a desire for further coverage of observational studies (comment 2), and wonders whether observational studies might suffer an even worse fate than trials (comment 2). I would like more research on observational studies as well, but as Song and colleagues note in their 2010 systematic review:

“Empirical studies on publication and related biases have focused mainly on certain areas of research such as clinical trials of health-care interventions. There is only very limited evidence on publication bias in many other research fields including basic research and observational studies.”[ref-2]^(p36)

Although the data are limited, there are some investigations into the matter, although many actually include a mixture of study types.[ref-2]^,[ref-11]^,[ref-27]^-[ref-30] When considering Ms. Bowles’ curiosity as to whether observational studies suffer even more from publication bias, the limitations in the available data ultimately preclude one from making any definitive statements, and the data we do have are not congruent. In a subgroup analysis, Easterbrook and colleagues found observational and laboratory-based experimental studies considered together may be more susceptible to publication bias than randomized controlled trials.[ref-29] However, Dickersin and colleagues did not detect any difference in clinical trials versus other studies,[ref-11] and Stern and Simes found the risk of publication bias was even stronger for the subgroup of clinical trials included in their study compared to all the studies considered together.[ref-30] Returning to the 2010 systematic review, Song and colleagues note the results on publication bias were materially indifferent when analyses were restricted to clinical trials versus including all studies.[ref-2]

Although observational data are certainly useful, it seems at least possible that the focus on clinical trials is due to clinical trials representing a higher, more reliable level of evidence when making inferences about an intervention’s effects, a sentiment echoed by Dwan and colleagues in their 2013 systematic review.[ref-27] It also seems at least feasible that this is why efforts to improve the situation have focused on clinical trials. This certainly is not to say observational data are unimportant; indeed, we should strive to gain access to all data, not just data from trials. Perhaps the hopeful success of efforts currently underway for trials can be used as a framework for other study types; in the meantime, however, operating within the existing frameworks and efforts and within the confines of reality, it seems reasonable and important to continue with the efforts focused on clinical trials, understanding that no sensible person would consider gaining proper access to untainted clinical trial data as being the end of this plight if there is still evidence of issues with other study types. The end goal is, as Silverman opined, to be able to truly “narrow the area of uncertainty” as much as possible to build “working truths … that support our everyday actions at the bedside.”[ref-31]^(p165) As long as issues with publication bias and inadequate research transparency persist, this goal remains elusive, and we must therefore continue to fight until these issues are a sordid tale in our past.

Lastly, Ms. Bowles comments that I do not mention the impact that publication bias has on future research, specifically that missing research may result in research being unknowingly repeated, thereby potentially leading to wasting time and money (comment 4). However, I actually do hint at this indirectly in the last sentence of my introduction (“These issues distort the evidence base from which medical providers and researchers should be operating, and therefore put patients at risk of direct or indirect harm.”) and the second-to-last sentence of my conclusion (“Indeed, publication bias and inadequate research transparency represent a gaping wound in the body of evidence from which researchers operate and medical providers make decisions regarding care.”). I also more directly state the effects on research time and funding earlier in my conclusion:

“Continuing to tolerate these problems is not only an ethical blemish of the worst kind, but is also: wholly disrespectful to those who participate in clinical trials with the belief their participation will help improve care; a cause of inefficient use of research time and funding; and perhaps most importantly, a threat to providers’ ability to provide their patients with the best evidence-based care possible.”

Still, I thank Ms. Bowles for her comment here, as I do think I could have been more explicit when discussing how these issues can lead to inefficient use of research time and funding, and I think being more explicit would have made for an even stronger overall statement of the multifaceted deleterious effects these issues have.

I wholeheartedly agree with Ms. Bowles that the NIH should heavily penalize authors who fail to publish properly, and I honestly think this penalization should be extended beyond the NIH. Indeed, it seems monitoring and penalties (that are actually enforced) will be a necessary step in improving these issues. This speaks to part of the reason why I discuss in such detail how the government has failed to enforce Section 801 of the Food and Drug Administration Amendments Act of 2007, how the changes proposed in 2014 (including changes specific to trials receiving funding from the NIH) still fail to address enforcement and do not seem to offer much hope of true improvement, how the ICMJE proclamation has also not been upheld, and how current efforts underway seek to improve the state of affairs.

In summary: The available evidence, while not able to exculpate journals/editors entirely, still suggests authors/researchers are more to blame for publication bias than journals/editors; journals/editors seem willing to publish “negative”/non-significant results; we need more investigation into how these issues affect observational studies, but it is reasonable and important to continue with efforts focused on clinical trials for the time being (understanding that the fight will continue until publication bias and inadequate research transparency are truly a thing of the past); and these issues can lead to substantial wasting of time and funding due to unawareness of previously-conducted studies that were not properly disseminated. As I state in my article, previous measures have failed to rectify the issues of publication bias and inadequate research transparency (though these measures could prove meaningful with logistical changes, better oversight, and actual enforcement), current movements have gained vigor, and one hopes they will not fall short like their predecessors. Again, we must continue to pursue these matters until true resolution occurs; anything less than that is unacceptable and disgraceful.

[References]

[[1|type=journal|authors=Emerson/GB;Warme/WJ;Wolf/FM;Heckman/JD;Brand/RA;Leopold/SS|title=Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial|source=Arch Intern Med|year=2010|vol=170|issue=21|fpage=1934|lpage=1939|doi=10.1001/archinternmed.2010.406|pmid=21098355]][[2|type=journal|authors=Song/F;Parekh/S;Hooper/L;Loke/YK;Ryder/J;Sutton/AJ;Hing/C;Kwok/CS;Pang/C;Harvey/I|title=Dissemination and publication of research findings: an updated review of related biases|source=Health Technol Assess|year=2010|vol=14|issue=8|doi=10.3310/hta14080|pmid=20181324]][[3|type=journal|authors=Olson/CM;Rennie/D;Cook/D;Dickersin/K;Flanagin/A;Hogan/JW;Zhu/Q;Reiling/J;Pace/B|title=Publication bias in editorial decision making|source=JAMA|year=2002|vol=287|issue=21|fpage=2825|lpage=2828|pmid=12038924]]
[[4|type=journal|authors=Lee/KP;Boyd/EA;Holroyd-Leduc/JM;Bacchetti/P;Bero/LA|title=Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals|source=Med J Aust|year=2006|vol=184|issue=12|fpage=621|lpage=626|pmid=16803442]][[5|type=journal|authors=Lynch/JR;Cunningham/MRA;Warme/WJ;Schaad/DC;Wolf/FM;Leopold/SS|title=Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not|source=J Bone Joint Surg Am|year=2007|vol=89|issue=5|fpage=1010|lpage=1018|pmid=17473138]][[6|type=journal|authors=Okike/K;Kocher/MS;Mehlman/CT;Heckman/JD;Bhandari/M|title=Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume)|source=J Bone Joint Surg Am|year=2008|vol=90|issue=3|fpage=595|lpage=601|doi=10.2106/JBJS.G.00279|pmid=18310710]]
[[7|type=journal|authors=van Lent/M;Overbeke/J;Out/HJ|title=Role of editorial and peer review processes in publication bias: analysis of drug trials submitted to eight medical journals|source=PLoS One|year=2014|vol=9|issue=8|fpage=e104846|doi=10.1371/journal.pone.0104846|pmid=25118182|pmcid=4130599]]
[[8|type=journal|authors=Ioannidis/JPA|title=Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials|source=JAMA|year=1998|vol=279|issue=4|fpage=281|lpage=286|pmid=9450711]][[9|type=journal|authors=Dickersin/K;Olson/CM;Rennie/D;Cook/D;Flanagin/A;Zhu/Q;Reiling/J;Pace/B|title=Association between time interval to publication and statistical significance|source=JAMA|year=2002|vol=287|issue=21|fpage=2829|lpage=2831|pmid=12038925]][[10|type=journal|authors=Dickersin/K;Chan/S;Chalmers/TC;Sacks/HS;Smith/H Jr|title=Publication bias and clinical trials|source=Control Clin Trials|year=1987|vol=8|issue=4|fpage=343|lpage=353|pmid=3442991]]
[[11|type=journal|authors=Dickersin/K;Min/Y-I;Meinert/CL|title=Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards|source=JAMA|year=1992|vol=267|issue=3|fpage=374|lpage=378|pmid=1727960]]
[[12|type=journal|authors=Dickersin/K;Min/YI|title=Publication bias: the problem that won’t go away|source=Ann N Y Acad Sci|year=1993|vol=703|fpage=135|lpage=148|pmid=8192291]][[13|type=journal|authors=Weber/EJ;Callaham/ML;Wears/RL;Barton/C;Young/G|title=Unpublished research from a medical specialty meeting: why investigators fail to publish|source=JAMA|year=1998|vol=280|issue=3|fpage=257|lpage=259|pmid=9676674]][[14|type=journal|authors=Timmer/A;Hilsden/RJ;Cole/J;Hailey/D;Sutherland/LR|title=Publication bias in gastroenterological research - a retrospective cohort study based on abstracts submitted to a scientific meeting|source=BMC Med Res Methodol|year=2002|vol=2|fpage=7|pmid=11978183|pmcid=115863]]
[[15|type=journal|authors=Krzyzanowska/MK;Pintilie/M;Tannock/IF|title=Factors associated with failure to publish large randomized trials presented at an oncology meeting|source=JAMA|year=2003|vol=290|issue=4|fpage=495|lpage=501|pmid=12876092]]
[[16|type=journal|authors=Scherer/RW;Langenberg/P|title=Full publication of results initially presented in abstracts|source=Cochrane Database Syst Rev|year=2007|vol=18|issue=2|fpage=MR000005|pmid=17443628]]
[[17|authors=US National Insitutes of Health|title=Frequently Asked Questions|source=ClinicalTrials.gov|fpage=Last reviewed February 2015|lpage=Accessed October 11, 2015|url=https://www.clinicaltrials.gov/ct2/manage-recs/faq]]
[[18|type=journal|authors=Schleussner/E;Kamin/G;Seliger/G;Rogenhofer/N;Ebner/S;Toth/B;Schenk/M;Henes/M;Bohlmann/MK;Fischer/T;Brosteanu/O;Bauersachs/R;Petroff/D;ETHIG II group|title=Low-molecular-weight heparin for women with unexplained recurrent pregnancy loss: a multicenter trial with a minimization randomization scheme|source=Ann Intern Med|year=2015|vol=162|issue=9|fpage=601|lpage=609|doi=10.7326/M14-2062|pmid=25938990]]
[[19|type=journal|authors=Bolland/MJ;Leung/W;Tai/V;Bastin/S;Gamble/GD;Grey/A;Reid/IR|title=Calcium intake and risk of fracture: systematic review|source=BMJ|year=2015|vol=351|fpage=h4580|doi=10.1136/bmj.h4580|pmid=26420387]]
[[20|title=Efficacy and safety of paracetamol for spinal pain and osteoarthritis: systematic review and meta-analysis of randomised placebo controlled trials|authors=Machado/GC;Maher/CG;Ferreira/PH;Pinheiro/MB;Lin/CW;Day/RO;McLachlan/AJ;Ferreira/ML|source=BMJ|vol=350|issue=h1225|year=2015|type=journal|doi=10.1136/bmj.h1225|pmid=25828856|pmcid=PMC4381278|url=http://www.bmj.com/content/350/bmj.h1225.long]]
[[21|title=Effect of omega-3 fatty acids, lutein/zeaxanthin, or other nutrient supplementation on cognitive function: the AREDS2 randomized clinical trial|authors=Chew/EY;Clemons/TE;Agrón/E;Launer/LJ;Grodstein/F;Bernstein/PS;Age-Related Eye Disease Study 2 (AREDS2) Research Group|source=JAMA|vol=314|issue=8|year=2015|fpage=791|lpage=801|type=journal|doi=10.1001/jama.2015.9677|pmid=26305649|url=http://jama.jamanetwork.com/article.aspx?articleid=2429713]]
[[22|title=Surgical vs nonsurgical treatment of adults with displaced fractures of the proximal humerus: the PROFHER randomized clinical trial|authors=Rangan/A;Handoll/H;Brealey/S;Jefferson/L;Keding/A;Martin/BC;Goodchild/L;Chuang/LH;Hewitt/C;Torgerson/D;PROFHER Trial Collaborators|source=JAMA|vol=313|issue=10|year=2015|fpage=1037|lpage=1047|type=journal|doi=10.1001/jama.2015.1629|pmid=25756440|pmcid=25756440|url=http://jama.jamanetwork.com/article.aspx?articleid=2190987]]
[[23|title=Methylprednisolone in patients undergoing cardiopulmonary bypass (SIRS): a randomised, double-blind, placebo-controlled trial|authors=Whitlock/RP;Devereaux/PJ;Teoh/KH;Lamy/A;Vincent/J;Pogue/J;Paparella/D;Sessler/DI;Karthikeyan/G;Villar/JC;Zuo/Y;Avezum/Á;Quantz/M;Tagarakis/GI;Shah/PJ;Abbasi/SH;Zheng/H;Pettit/S;Chrolavicius/S;Yusuf/S|source=Lancet|vol=386|issue=10000|year=2015|fpage=1243|lpage=1253|type=journal|doi= 10.1016/S0140-6736(15)00273-1|url=http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2815%2900273-1/abstract]]
[[24|title=Medical expulsive therapy in adults with ureteric colic: a multicentre, randomised, placebo-controlled trial|authors=Pickard/R;Starr/K;MacLennan/G;Lam/T;Thomas/R;Burr/J;McPherson/G;McDonald/A;Anson/K;N'Dow/J;Burgess/N;Clark/T;Kilonzo/M;Gillies/K;Shearer/K;Boachie/C;Cameron/S;Norrie/J;McClinton/S|source=Lancet|vol=386|issue=9991|year=2015|fpage=341|lpage=349|type=journal|doi= 10.1016/S0140-6736(15)60933-3|pmid=25998582|url=http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2815%2960933-3/abstract]]
[[25|title=Spironolactone for heart failure with preserved ejection fraction|authors=Pitt/B;Pfeffer/MA;Assmann/SF;Boineau/R;Anand/IS;Claggett/B;Clausell/N;Desai;AS;Diaz/R;Fleg/JL;Gordeev/I;Harty/B;Heitner/JF;Kenwood/CT;Lewis/EF;O'Meara/E;Probstfield/JL;Shaburishvili/T;Shah/SJ;Solomon/SD;Sweitzer/NK;Yang/S;McKinlay/SM;TOPCAT Investigators|source=N Engl J Med|vol=370|issue=15|year=2014|fpage=1383|lpage=1392|type=journal|doi=10.1056/NEJMoa1313731|pmid=24716680|url=http://www.nejm.org/doi/full/10.1056/NEJMoa1313731]]
[[26|title=Cyclosporine before PCI in Patients with Acute Myocardial Infarction|authors=Cung/TT;Morel/O;Cayla/G;Rioufol/G;Garcia-Dorado/D;Angoulvant/D;Bonnefoy-Cudraz/E;Guérin/P;Elbaz/M;Delarche/N;Coste/P;Vanzetto/G;Metge/M;Aupetit/J-F;Jouve/B;Motreff/P;Tron/C;Labeque/J-N;Steg/PG;Cottin/Y; Range/G;Clerc/J;Claeys/MJ;Coussement/P;Prunier/F;Moulin/F;Roth/M;Belle/L;Dubois/P;Barragan/P;Gilard/M;Piot/C;Colin/P;De Poli/F;Morice/M-C;Ider/O;Dubois-Randé/J-L;Unterseeh/T;Le Breton/H;Béard/T;Blanchard/D;Grollier/G;Malquarti/V;Staat/P;Sudre/A;Elmer/E;Hansson/MJ;Bergerot/C;Boussaha/I;Jossan/C;Derumeaux/G;Mewton/N;Michel/O|source=N Engl J Med|vol=373|year=2015|fpage=1021|lpage=1031|type=journal|doi=10.1056/NEJMoa1505489|url=http://www.nejm.org/doi/full/10.1056/NEJMoa1505489]]
[[27|title=Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review|authors=Dwan/K;Altman/DG;Arnaiz/JA;Bloom/J;Chan/AW;Cronin/E;Decullier/E;Easterbrook/PJ;Von Elm/E;Gamble/C;Ghersi/D;Ioannidis/JP;Simes/J;Williamson/PR|source=PLoS One|vol=3|issue=8|year=2008|fpage=e3081|type=journal|doi=10.1371/journal.pone.0003081|pmid=18769481|pmcid=PMC2518111|url=http://www.ncbi.nlm.nih.gov/pubmed/18769481]]
[[28|type=journal|authors=Kavvoura/FK;Liberopoulos/G;Ioannidis/JP|title=Selection in reported epidemiological risks: an empirical assessment|source=PLoS Med|year=2007|vol=4|issue=3|fpage=e79|pmid=17341129 |pmcid=1808481]]
[[29|type=journal|authors=Easterbrook/PJ;Berlin/JA;Gopalan/R;Matthews/DR|title=Publication bias in clinical research|source=Lancet|year=1991|vol=337|issue=8746|fpage=867|lpage=872|pmid=1672966]]
[[30|title=Publication bias: evidence of delayed publication in a cohort study of clinical research projects|authors=Stern/JM;Simes/RJ|source=BMJ|vol=315|issue=7109|year=1997|fpage=640|lpage=645|type=journal|pmid=9310565|url=http://www.bmj.com/content/315/7109/640]]
[[31|title=Where's the Evidence?: Debates in Modern Medicine|authors=Silverman/WA|vol=Oxford University Press|issue=Oxford|year=1999|type=book|url=http://ukcatalogue.oup.com/product/9780192630889.do#]]
I am glad Ms. Bowles appreciates the gravity of these issues, and I am grateful for her overall positive review of my article. My article was not designed as a primary research article, and therefore there were no primary scientific data to report. Still, although this does mean one cannot judge scientific quality in terms of methodological appraisal for a primary research article (or a meta-analysis, etc.), I would nevertheless posit my article can still be judged as a positive contribution to the biomedical sciences, as one can ultimately think of science in two broad, interrelated senses: Firstly, science is ultimately a way of thinking about and approaching phenomena that involves asking answerable questions and then systematically and impartially pursuing and using the totality of the best-available evidence to reach an informed and reasonable stance, a stance that should remain appropriately malleable as new evidence accumulates and gets incorporated into the current body of knowledge on a subject; secondly, science is a collaborative endeavor to achieve such goals, to collegially seek out such working truths as best we can.

This collaborative aspect speaks to my sincere appreciation of Ms. Bowles’ review and comments regarding my article. Below, I respond to her comments at length (with the comment[s] to which I am responding indicated in the responses).

Ms. Bowles comments that much attention is paid to ClinicalTrials.gov, but that biases from journals may have been overlooked, and that she believes this is where much of the bias lies (comment 1); she is also concerned that the willingness of certain journals to publish “null results” might have been overlooked, and that this could be part of the solution (comment 3). To be sure, however, no such oversights occurred; although I originally decided to omit an in-depth discussion of such matters in my article for the sake of length and for reasons that will become clear as I respond to these comments, I will gladly expound here, and the available evidence does offer some reassurance and illumination, though it does not support Ms. Bowles’ comments.

Although bias in the peer-review process has been demonstrated,¹ the best-available evidence overall suggests the majority of the problem actually rests with researchers/authors failing to publish or otherwise make available the results of their research, not journals/editors preferentially publishing only studies with “positive”/significant findings.²^-¹⁴ Indeed, this is discussed in the 2010 systematic review I cite in my article, which pooled the four studies to date that had investigated this matter.² Song and colleagues found a pooled odds ratio of acceptance of 1.06 for manuscripts with “positive”/significant findings versus manuscripts with “negative”/non-significant findings (95% confidence interval [CI], 0.80 to 1.39), which they rightly conclude as corroborating the notion that acceptance of a submitted manuscript was not significantly associated with whether the paper had “positive”/significant or “negative”/non-significant findings. The four studies included in the pooled analysis included two studies investigating general medical journals (JAMA in one study,³ and The BMJ, The Lancet, and Annals of Internal Medicine in the other⁴) and two studies investigating the American volume of the Journal of Bone and Joint Surgery.⁵^,⁶ All these studies individually arrived at the same result as well: once submitted to a journal for consideration, manuscripts with “positive”/significant findings are no more likely to be published than manuscripts with “negative”/non-significant findings.

One of the studies⁵ – which investigated only manuscripts on hip or knee arthroplasty submitted to the American volume of the Journal of Bone and Joint Surgery – has a potentially suggestive title, and it drew attention to the significant difference in the average quality of “positive”/significant manuscripts versus “negative”/non-significant manuscripts when quality was assessed by a score derived by the authors from standard criteria used to assess the quality of a piece of evidence (termed the Sackett score by the authors). On average, “negative”/non-significant manuscripts had higher Sackett scores than “positive”/significant manuscripts. When considering the mean level of evidence scale officially required by the journal during manuscript preparation and submission, however, there was no significant difference between “positive”/significant manuscripts and “negative”/non-significant manuscripts. There was no difference in acceptance based on study outcome, and intriguingly, there was no difference in acceptance based on study quality as assessed by either the mean Sackett score or the mean level of evidence scale officially used and required by the journal. This finding causes one to ponder, as one would rightly expect higher-quality manuscripts to be published with greater frequency than lower-quality manuscripts. They discuss the finding of no difference in quality between accepted versus rejected manuscripts thusly:

“Potential explanations for this finding include the possibility that the overall quality of submissions to The Journal is so high that quality differences among them are too subtle to be discerned with use of standard evidence-based-medicine scoring approaches or, more likely, that reviewers make their decisions on the basis of other criteria, such as timeliness, economic impact, or perceived relevance, in addition to those recommended in standard evidence-based-medicine texts.”⁵^(p1016)

However, these postulated explanations still leave one with at least some degree of lingering uncertainty regarding what to think about this finding and how to reconcile this finding with the other assessments and discussions of quality in this study. The authors also acknowledge that assessing manuscript quality and any relationship with acceptance versus rejection as being “outside the scope of [the] primary study end points as defined by [their] a priori hypotheses” and that “until or unless this finding is validated by others it should not be accorded great weight.” ⁵^(p1016) (Indeed, investigating study quality was not a part of either of the two clearly-stated primary research hypotheses.)

Furthermore, a large weakness of this study is its lack of attempt to control for potentially influential or confounding variables. Considering all of this, one should be cautious about making any strong conclusions about any findings of manuscript quality or relationships pertaining to manuscript quality and publication from this study, and even setting all this aside, one must also remember this study looked at submissions to one specialty journal and included only studies on hip and knee arthroplasty. Even the subsequent study investigating submissions to the American volume of the Journal of Bone and Joint Surgery noted that, due to the other study’s limitations, “it remains unclear whether publication bias exists in the orthopaedic journal editorial review process.”⁶^(p596) The subsequent study thus set its primary research intent as seeking to determine whether manuscripts with “positive”/significant findings were more likely to be published after controlling for various quality measures, level of evidence, and subspecialty field.⁶ In this study and the two studies that investigated general medical journals³^,⁴ – all of which investigated quality measures and utilized multivariate analyses to control for potentially influential or confounding variables – no difference was found in publication rates for submitted manuscripts with “positive”/significant results versus submitted manuscripts with “negative”/non-significant results. These studies also found that manuscripts of higher quality or level of evidence were significantly more likely to be published.³^,⁴^,⁶

Recently, a retrospective review of manuscripts from several journals (The BMJ, Annals of the Rheumatic Diseases, British Journal of Ophthalmology, Gut, Heart, Thorax, Diabetologia, and Journal of Hepatology) was published in PLoS ONE,⁷ and although retrospective studies are often viewed as having potential weaknesses compared to prospective studies, the retrospective design offers a unique benefit in this case: the elimination of any Hawthorne effect. Such an effect might occur in the aforementioned prospective studies on this issue, since editors and reviewers might, whether consciously or unconsciously, behave somewhat differently if they are aware their actions are being studied. This retrospective study also found no difference in acceptance for publication for submitted manuscripts with “positive”/significant findings compared to submitted manuscripts with “negative”/non-significant findings, this study considered quality indicators as well, and the finding of no difference was present in both the univariate and multivariate analyses.

However, what if merely looking at publication in a binary manner misses a level of nuance that might still have an impact? What if journals/editors are less enthusiastic about publishing “negative”/non-significant results compared to “positive”/significant results, and therefore, whether unconsciously or consciously, do not pursue the steps to publication as diligently, leading to significant delays in publication? Ioannidis found that, among a cohort of 66 completed multicenter trials focused on human immunodeficiency virus, the 45 trials that had been submitted for publication were published slightly more quickly after submission if they had “positive”/significant results (median of 0.8 years versus 1.1 years; P = 0.04); however, the ultimate clinical significance of an approximate 3.6 month delay is unclear.⁸ However, in another study investigating a cohort of 133 trials submitted to JAMA, there was no difference in time from submission to publication for “negative”/non-significant results compared to “positive”/significant results (median of 7.6 months versus 7.8 months, respectively; P = 0.44).⁹

Of course, a potential limitation in these studies is the inherent inability to account for manuscripts/studies that are not even written up and submitted to journals. The theoretical question of what these results would look like if non-submitted manuscripts were submitted is certainly an interesting question, even if only for the sake of refining our understanding of this matter. In the absence of such, however, the aforementioned studies remain our best evidence on the matter, and it would be entirely unfair to hold any speculative counterfactual against the journals/editors.

In summary, when considering what is known about publication bias and what this evidence shows, while the evidence may not allow us to confidently exculpate journals/editors entirely, it surely corroborates the notion that authors/researchers are more culpable than journals/editors.

Further insight can be gained from asking the researchers themselves why they failed to write up and submit a study.²^,¹⁰^-¹⁶ Here too, we see the majority of the problem seems to rest not with the journals/editors, but with the authors/researchers, with the most common reasons for not writing up and submitting a study tending to be lack of time or lack of interest. Both reasons are wholly unacceptable. Being rejected by a journal actually tends to be reported by only a minority, but this too is an unacceptable excuse; manuscript rejection is largely to be expected at some point in one’s publishing endeavors (regardless of the manuscript’s findings/content), and allowing this to be a deterrent is a fault of the researchers/authors, not the journals/editors. Due to cognitive biases, authors/researchers may be more likely to assign fault or biased motives to the journal/editors (and thus may also view things in a more emotionally charged manner) when a manuscript with “negative”/non-significant results is rejected compared to when a manuscript with “positive”/significant results is rejected; however, this is merely a speculative suggestion, albeit one that would seem to fit within what we know about human behavior. In any case, one must remember authors/researchers are specifically called to adhere to the World Medical Association’s Declaration of Helsinki in making the results of their research on humans publicly available (see the second sentence of principle 36 in Figure 2 in my article). Furthermore, at least for clinical trials, ClinicalTrials.gov could serve as a repository for study methods and results even if the authors/researchers do not wish to seek traditional publication or get rejected and decide not to pursue publication elsewhere (although my article discusses mandatory submission, ClinicalTrials.gov also accepts voluntary submissions).¹⁷ Importantly, this trumps any excuse pertaining to journal/editorial rejection.

On the note of ClinicalTrials.gov and the length to which I discuss it (comment 1), I do so because ClinicalTrials.gov was ushered in with great fanfare, and yet it has unequivocally failed and was doomed at the outset to be a partial success at best in its original (and current) form. Such assertions are admittedly bold, so the space I devote to dissecting this issue was seen as necessary, and truthfully, the space I devote to this topic is ultimately relatively limited given the evidence on the matter. However, the failure of ClinicalsTrials.gov to rectify the issues of publication bias and inadequate research transparency does not preclude it from being used in the aforementioned manner; moreover, with some important logistical changes and better oversight and enforcement (all of which I address in my article), ClinicalTrials.gov could actually be a remarkable tool in the fight to end publication bias and inadequate research transparency.

Further questioning the role the journals/editors might play, Ms. Bowles also questions the willingness of a top-tier journal to publish non-significant results, asking when a top-tier journal last published a non-significant result (comment 1, the response to which will also further address comment 3). However, there are several recent examples of studies with “null,” “negative,” or “non-significant” results even within the so-called “Big 5,” including (in alphabetic order) Annals of Internal Medicine,¹⁸ The BMJ,¹⁹^,²⁰Journal of the American Medical Association,²¹^,²² The Lancet,²³^,²⁴ and New England Journal of Medicine.²⁵^,²⁶

Again, considering all of this, although the evidence cannot entirely rule out journal/editorial bias, the totality of the best-available evidence nevertheless suggests the majority of the problem of publication bias rests more with authors/researchers, not with journals/editors. However, as I discuss clearly in my article, journals/editors are not blameless; indeed, a large area of failure has been the disappointing failure to uphold the ICMJE proclamation.

Ms. Bowles also comments on the seeming focus on clinical trials (comment 1), expresses a desire for further coverage of observational studies (comment 2), and wonders whether observational studies might suffer an even worse fate than trials (comment 2). I would like more research on observational studies as well, but as Song and colleagues note in their 2010 systematic review:

“Empirical studies on publication and related biases have focused mainly on certain areas of research such as clinical trials of health-care interventions. There is only very limited evidence on publication bias in many other research fields including basic research and observational studies.”²^(p36)

Although the data are limited, there are some investigations into the matter, although many actually include a mixture of study types.²^,¹¹^,²⁷^-³⁰ When considering Ms. Bowles’ curiosity as to whether observational studies suffer even more from publication bias, the limitations in the available data ultimately preclude one from making any definitive statements, and the data we do have are not congruent. In a subgroup analysis, Easterbrook and colleagues found observational and laboratory-based experimental studies considered together may be more susceptible to publication bias than randomized controlled trials.²⁹ However, Dickersin and colleagues did not detect any difference in clinical trials versus other studies,¹¹ and Stern and Simes found the risk of publication bias was even stronger for the subgroup of clinical trials included in their study compared to all the studies considered together.³⁰ Returning to the 2010 systematic review, Song and colleagues note the results on publication bias were materially indifferent when analyses were restricted to clinical trials versus including all studies.²

Although observational data are certainly useful, it seems at least possible that the focus on clinical trials is due to clinical trials representing a higher, more reliable level of evidence when making inferences about an intervention’s effects, a sentiment echoed by Dwan and colleagues in their 2013 systematic review.²⁷ It also seems at least feasible that this is why efforts to improve the situation have focused on clinical trials. This certainly is not to say observational data are unimportant; indeed, we should strive to gain access to all data, not just data from trials. Perhaps the hopeful success of efforts currently underway for trials can be used as a framework for other study types; in the meantime, however, operating within the existing frameworks and efforts and within the confines of reality, it seems reasonable and important to continue with the efforts focused on clinical trials, understanding that no sensible person would consider gaining proper access to untainted clinical trial data as being the end of this plight if there is still evidence of issues with other study types. The end goal is, as Silverman opined, to be able to truly “narrow the area of uncertainty” as much as possible to build “working truths … that support our everyday actions at the bedside.”³¹^(p165) As long as issues with publication bias and inadequate research transparency persist, this goal remains elusive, and we must therefore continue to fight until these issues are a sordid tale in our past.

Lastly, Ms. Bowles comments that I do not mention the impact that publication bias has on future research, specifically that missing research may result in research being unknowingly repeated, thereby potentially leading to wasting time and money (comment 4). However, I actually do hint at this indirectly in the last sentence of my introduction (“These issues distort the evidence base from which medical providers and researchers should be operating, and therefore put patients at risk of direct or indirect harm.”) and the second-to-last sentence of my conclusion (“Indeed, publication bias and inadequate research transparency represent a gaping wound in the body of evidence from which researchers operate and medical providers make decisions regarding care.”). I also more directly state the effects on research time and funding earlier in my conclusion:

“Continuing to tolerate these problems is not only an ethical blemish of the worst kind, but is also: wholly disrespectful to those who participate in clinical trials with the belief their participation will help improve care; a cause of inefficient use of research time and funding; and perhaps most importantly, a threat to providers’ ability to provide their patients with the best evidence-based care possible.”

Still, I thank Ms. Bowles for her comment here, as I do think I could have been more explicit when discussing how these issues can lead to inefficient use of research time and funding, and I think being more explicit would have made for an even stronger overall statement of the multifaceted deleterious effects these issues have.

I wholeheartedly agree with Ms. Bowles that the NIH should heavily penalize authors who fail to publish properly, and I honestly think this penalization should be extended beyond the NIH. Indeed, it seems monitoring and penalties (that are actually enforced) will be a necessary step in improving these issues. This speaks to part of the reason why I discuss in such detail how the government has failed to enforce Section 801 of the Food and Drug Administration Amendments Act of 2007, how the changes proposed in 2014 (including changes specific to trials receiving funding from the NIH) still fail to address enforcement and do not seem to offer much hope of true improvement, how the ICMJE proclamation has also not been upheld, and how current efforts underway seek to improve the state of affairs.

In summary: The available evidence, while not able to exculpate journals/editors entirely, still suggests authors/researchers are more to blame for publication bias than journals/editors; journals/editors seem willing to publish “negative”/non-significant results; we need more investigation into how these issues affect observational studies, but it is reasonable and important to continue with efforts focused on clinical trials for the time being (understanding that the fight will continue until publication bias and inadequate research transparency are truly a thing of the past); and these issues can lead to substantial wasting of time and funding due to unawareness of previously-conducted studies that were not properly disseminated. As I state in my article, previous measures have failed to rectify the issues of publication bias and inadequate research transparency (though these measures could prove meaningful with logistical changes, better oversight, and actual enforcement), current movements have gained vigor, and one hopes they will not fall short like their predecessors. Again, we must continue to pursue these matters until true resolution occurs; anything less than that is unacceptable and disgraceful.

References
1. Emerson GB, Warme WJ, Wolf FM, Heckman JD, et al.: Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Arch Intern Med. 2010; 170 (21): 1934-1939 PubMed Abstract | Publisher Full Text
2. Song F, Parekh S, Hooper L, Loke YK, et al.: Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010; 14 (8). PubMed Abstract | Publisher Full Text
3. Olson CM, Rennie D, Cook D, Dickersin K, et al.: Publication bias in editorial decision making. JAMA. 2002; 287 (21): 2825-2828 PubMed Abstract
4. Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, et al.: Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006; 184 (12): 621-626 PubMed Abstract
5. Lynch JR, Cunningham MRA, Warme WJ, Schaad DC, et al.: Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. J Bone Joint Surg Am. 2007; 89 (5): 1010-1018 PubMed Abstract
6. Okike K, Kocher MS, Mehlman CT, Heckman JD, et al.: Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008; 90 (3): 595-601 PubMed Abstract | Publisher Full Text
7. van Lent M, Overbeke J, Out HJ: Role of editorial and peer review processes in publication bias: analysis of drug trials submitted to eight medical journals. PLoS One. 2014; 9 (8): e104846 PubMed Abstract | Free Full Text | Publisher Full Text
8. Ioannidis JPA: Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA. 1998; 279 (4): 281-286 PubMed Abstract
9. Dickersin K, Olson CM, Rennie D, Cook D, et al.: Association between time interval to publication and statistical significance. JAMA. 2002; 287 (21): 2829-2831 PubMed Abstract
10. Dickersin K, Chan S, Chalmers TC, Sacks HS, et al.: Publication bias and clinical trials. Control Clin Trials. 1987; 8 (4): 343-353 PubMed Abstract
11. Dickersin K, Min Y-I, Meinert CL: Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 1992; 267 (3): 374-378 PubMed Abstract
12. Dickersin K, Min YI: Publication bias: the problem that won’t go away. Ann N Y Acad Sci. 1993; 703: 135-148 PubMed Abstract
13. Weber EJ, Callaham ML, Wears RL, Barton C, et al.: Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA. 1998; 280 (3): 257-259 PubMed Abstract
14. Timmer A, Hilsden RJ, Cole J, Hailey D, et al.: Publication bias in gastroenterological research - a retrospective cohort study based on abstracts submitted to a scientific meeting. BMC Med Res Methodol. 2002; 2: 7 PubMed Abstract | Free Full Text
15. Krzyzanowska MK, Pintilie M, Tannock IF: Factors associated with failure to publish large randomized trials presented at an oncology meeting. JAMA. 2003; 290 (4): 495-501 PubMed Abstract
16. Scherer RW, Langenberg P: Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007; 18 (2): MR000005 PubMed Abstract
17. US National Insitutes of Health: Frequently Asked Questions. ClinicalTrials.gov. Last reviewed February 2015-Accessed October 11, 2015 Reference Source
18. Schleussner E, Kamin G, Seliger G, Rogenhofer N, et al.: Low-molecular-weight heparin for women with unexplained recurrent pregnancy loss: a multicenter trial with a minimization randomization scheme. Ann Intern Med. 2015; 162 (9): 601-609 PubMed Abstract | Publisher Full Text
19. Bolland MJ, Leung W, Tai V, Bastin S, et al.: Calcium intake and risk of fracture: systematic review. BMJ. 2015; 351: h4580 PubMed Abstract | Publisher Full Text
20. Machado GC, Maher CG, Ferreira PH, Pinheiro MB, et al.: Efficacy and safety of paracetamol for spinal pain and osteoarthritis: systematic review and meta-analysis of randomised placebo controlled trials. BMJ. 2015; 350 (h1225). PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
21. Chew EY, Clemons TE, Agrón E, Launer LJ, et al.: Effect of omega-3 fatty acids, lutein/zeaxanthin, or other nutrient supplementation on cognitive function: the AREDS2 randomized clinical trial. JAMA. 2015; 314 (8): 791-801 PubMed Abstract | Publisher Full Text | Reference Source
22. Rangan A, Handoll H, Brealey S, Jefferson L, et al.: Surgical vs nonsurgical treatment of adults with displaced fractures of the proximal humerus: the PROFHER randomized clinical trial. JAMA. 2015; 313 (10): 1037-1047 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
23. Whitlock RP, Devereaux PJ, Teoh KH, Lamy A, et al.: Methylprednisolone in patients undergoing cardiopulmonary bypass (SIRS): a randomised, double-blind, placebo-controlled trial. Lancet. 2015; 386 (10000): 1243-1253 Publisher Full Text | Reference Source
24. Pickard R, Starr K, MacLennan G, Lam T, et al.: Medical expulsive therapy in adults with ureteric colic: a multicentre, randomised, placebo-controlled trial. Lancet. 2015; 386 (9991): 341-349 PubMed Abstract | Publisher Full Text | Reference Source
25. Pitt B, Pfeffer MA, Assmann SF, Boineau R, et al.: Spironolactone for heart failure with preserved ejection fraction. N Engl J Med. 2014; 370 (15): 1383-1392 PubMed Abstract | Publisher Full Text | Reference Source
26. Cung TT, Morel O, Cayla G, Rioufol G, et al.: Cyclosporine before PCI in Patients with Acute Myocardial Infarction. N Engl J Med. 2015; 373: 1021-1031 Publisher Full Text | Reference Source
27. Dwan K, Altman DG, Arnaiz JA, Bloom J, et al.: Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review. PLoS One. 2008; 3 (8): e3081 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
28. Kavvoura FK, Liberopoulos G, Ioannidis JP: Selection in reported epidemiological risks: an empirical assessment. PLoS Med. 2007; 4 (3): e79 PubMed Abstract | Free Full Text
29. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991; 337 (8746): 867-872 PubMed Abstract
30. Stern JM, Simes RJ: Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997; 315 (7109): 640-645 PubMed Abstract | Reference Source
31. Silverman WA: Where's the Evidence?: Debates in Modern Medicine. 1999; Oxford University Press (Oxford). Reference Source
Competing Interests: Please see my original competing interests statement, which remains in effect and unchanged since publishing my original article. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 13 Oct 2015

Martin Mayer, Department of Physician Assistant Studies, College of Allied Health Sciences, East Carolina University, Greenville, 27858-4353, USA

13 Oct 2015

Author Response

I am glad Ms. Bowles appreciates the gravity of these issues, and I am grateful for her overall positive review of my article. My article was not designed as a ... Continue reading I am glad Ms. Bowles appreciates the gravity of these issues, and I am grateful for her overall positive review of my article. My article was not designed as a primary research article, and therefore there were no primary scientific data to report. Still, although this does mean one cannot judge scientific quality in terms of methodological appraisal for a primary research article (or a meta-analysis, etc.), I would nevertheless posit my article can still be judged as a positive contribution to the biomedical sciences, as one can ultimately think of science in two broad, interrelated senses: Firstly, science is ultimately a way of thinking about and approaching phenomena that involves asking answerable questions and then systematically and impartially pursuing and using the totality of the best-available evidence to reach an informed and reasonable stance, a stance that should remain appropriately malleable as new evidence accumulates and gets incorporated into the current body of knowledge on a subject; secondly, science is a collaborative endeavor to achieve such goals, to collegially seek out such working truths as best we can.

This collaborative aspect speaks to my sincere appreciation of Ms. Bowles’ review and comments regarding my article. Below, I respond to her comments at length (with the comment[s] to which I am responding indicated in the responses).

Ms. Bowles comments that much attention is paid to ClinicalTrials.gov, but that biases from journals may have been overlooked, and that she believes this is where much of the bias lies (comment 1); she is also concerned that the willingness of certain journals to publish “null results” might have been overlooked, and that this could be part of the solution (comment 3). To be sure, however, no such oversights occurred; although I originally decided to omit an in-depth discussion of such matters in my article for the sake of length and for reasons that will become clear as I respond to these comments, I will gladly expound here, and the available evidence does offer some reassurance and illumination, though it does not support Ms. Bowles’ comments.

Although bias in the peer-review process has been demonstrated,[ref-1] the best-available evidence overall suggests the majority of the problem actually rests with researchers/authors failing to publish or otherwise make available the results of their research, not journals/editors preferentially publishing only studies with “positive”/significant findings.[ref-2]^-[ref-14] Indeed, this is discussed in the 2010 systematic review I cite in my article, which pooled the four studies to date that had investigated this matter.[ref-2] Song and colleagues found a pooled odds ratio of acceptance of 1.06 for manuscripts with “positive”/significant findings versus manuscripts with “negative”/non-significant findings (95% confidence interval [CI], 0.80 to 1.39), which they rightly conclude as corroborating the notion that acceptance of a submitted manuscript was not significantly associated with whether the paper had “positive”/significant or “negative”/non-significant findings. The four studies included in the pooled analysis included two studies investigating general medical journals (JAMA in one study,[ref-3] and The BMJ, The Lancet, and Annals of Internal Medicine in the other[ref-4]) and two studies investigating the American volume of the Journal of Bone and Joint Surgery.[ref-5]^,[ref-6] All these studies individually arrived at the same result as well: once submitted to a journal for consideration, manuscripts with “positive”/significant findings are no more likely to be published than manuscripts with “negative”/non-significant findings.

One of the studies[ref-5] – which investigated only manuscripts on hip or knee arthroplasty submitted to the American volume of the Journal of Bone and Joint Surgery – has a potentially suggestive title, and it drew attention to the significant difference in the average quality of “positive”/significant manuscripts versus “negative”/non-significant manuscripts when quality was assessed by a score derived by the authors from standard criteria used to assess the quality of a piece of evidence (termed the Sackett score by the authors). On average, “negative”/non-significant manuscripts had higher Sackett scores than “positive”/significant manuscripts. When considering the mean level of evidence scale officially required by the journal during manuscript preparation and submission, however, there was no significant difference between “positive”/significant manuscripts and “negative”/non-significant manuscripts. There was no difference in acceptance based on study outcome, and intriguingly, there was no difference in acceptance based on study quality as assessed by either the mean Sackett score or the mean level of evidence scale officially used and required by the journal. This finding causes one to ponder, as one would rightly expect higher-quality manuscripts to be published with greater frequency than lower-quality manuscripts. They discuss the finding of no difference in quality between accepted versus rejected manuscripts thusly:

“Potential explanations for this finding include the possibility that the overall quality of submissions to The Journal is so high that quality differences among them are too subtle to be discerned with use of standard evidence-based-medicine scoring approaches or, more likely, that reviewers make their decisions on the basis of other criteria, such as timeliness, economic impact, or perceived relevance, in addition to those recommended in standard evidence-based-medicine texts.”[ref-5]^(p1016)

However, these postulated explanations still leave one with at least some degree of lingering uncertainty regarding what to think about this finding and how to reconcile this finding with the other assessments and discussions of quality in this study. The authors also acknowledge that assessing manuscript quality and any relationship with acceptance versus rejection as being “outside the scope of [the] primary study end points as defined by [their] a priori hypotheses” and that “until or unless this finding is validated by others it should not be accorded great weight.” [ref-5]^(p1016) (Indeed, investigating study quality was not a part of either of the two clearly-stated primary research hypotheses.)

Furthermore, a large weakness of this study is its lack of attempt to control for potentially influential or confounding variables. Considering all of this, one should be cautious about making any strong conclusions about any findings of manuscript quality or relationships pertaining to manuscript quality and publication from this study, and even setting all this aside, one must also remember this study looked at submissions to one specialty journal and included only studies on hip and knee arthroplasty. Even the subsequent study investigating submissions to the American volume of the Journal of Bone and Joint Surgery noted that, due to the other study’s limitations, “it remains unclear whether publication bias exists in the orthopaedic journal editorial review process.”[ref-6]^(p596) The subsequent study thus set its primary research intent as seeking to determine whether manuscripts with “positive”/significant findings were more likely to be published after controlling for various quality measures, level of evidence, and subspecialty field.[ref-6] In this study and the two studies that investigated general medical journals[ref-3]^,[ref-4] – all of which investigated quality measures and utilized multivariate analyses to control for potentially influential or confounding variables – no difference was found in publication rates for submitted manuscripts with “positive”/significant results versus submitted manuscripts with “negative”/non-significant results. These studies also found that manuscripts of higher quality or level of evidence were significantly more likely to be published.[ref-3]^,[ref-4]^,[ref-6]

Recently, a retrospective review of manuscripts from several journals (The BMJ, Annals of the Rheumatic Diseases, British Journal of Ophthalmology, Gut, Heart, Thorax, Diabetologia, and Journal of Hepatology) was published in PLoS ONE,[ref-7] and although retrospective studies are often viewed as having potential weaknesses compared to prospective studies, the retrospective design offers a unique benefit in this case: the elimination of any Hawthorne effect. Such an effect might occur in the aforementioned prospective studies on this issue, since editors and reviewers might, whether consciously or unconsciously, behave somewhat differently if they are aware their actions are being studied. This retrospective study also found no difference in acceptance for publication for submitted manuscripts with “positive”/significant findings compared to submitted manuscripts with “negative”/non-significant findings, this study considered quality indicators as well, and the finding of no difference was present in both the univariate and multivariate analyses.

However, what if merely looking at publication in a binary manner misses a level of nuance that might still have an impact? What if journals/editors are less enthusiastic about publishing “negative”/non-significant results compared to “positive”/significant results, and therefore, whether unconsciously or consciously, do not pursue the steps to publication as diligently, leading to significant delays in publication? Ioannidis found that, among a cohort of 66 completed multicenter trials focused on human immunodeficiency virus, the 45 trials that had been submitted for publication were published slightly more quickly after submission if they had “positive”/significant results (median of 0.8 years versus 1.1 years; P = 0.04); however, the ultimate clinical significance of an approximate 3.6 month delay is unclear.[ref-8] However, in another study investigating a cohort of 133 trials submitted to JAMA, there was no difference in time from submission to publication for “negative”/non-significant results compared to “positive”/significant results (median of 7.6 months versus 7.8 months, respectively; P = 0.44).[ref-9]

Of course, a potential limitation in these studies is the inherent inability to account for manuscripts/studies that are not even written up and submitted to journals. The theoretical question of what these results would look like if non-submitted manuscripts were submitted is certainly an interesting question, even if only for the sake of refining our understanding of this matter. In the absence of such, however, the aforementioned studies remain our best evidence on the matter, and it would be entirely unfair to hold any speculative counterfactual against the journals/editors.

In summary, when considering what is known about publication bias and what this evidence shows, while the evidence may not allow us to confidently exculpate journals/editors entirely, it surely corroborates the notion that authors/researchers are more culpable than journals/editors.

Further insight can be gained from asking the researchers themselves why they failed to write up and submit a study.[ref-2]^,[ref-10]^-[ref-16] Here too, we see the majority of the problem seems to rest not with the journals/editors, but with the authors/researchers, with the most common reasons for not writing up and submitting a study tending to be lack of time or lack of interest. Both reasons are wholly unacceptable. Being rejected by a journal actually tends to be reported by only a minority, but this too is an unacceptable excuse; manuscript rejection is largely to be expected at some point in one’s publishing endeavors (regardless of the manuscript’s findings/content), and allowing this to be a deterrent is a fault of the researchers/authors, not the journals/editors. Due to cognitive biases, authors/researchers may be more likely to assign fault or biased motives to the journal/editors (and thus may also view things in a more emotionally charged manner) when a manuscript with “negative”/non-significant results is rejected compared to when a manuscript with “positive”/significant results is rejected; however, this is merely a speculative suggestion, albeit one that would seem to fit within what we know about human behavior. In any case, one must remember authors/researchers are specifically called to adhere to the World Medical Association’s Declaration of Helsinki in making the results of their research on humans publicly available (see the second sentence of principle 36 in Figure 2 in my article). Furthermore, at least for clinical trials, ClinicalTrials.gov could serve as a repository for study methods and results even if the authors/researchers do not wish to seek traditional publication or get rejected and decide not to pursue publication elsewhere (although my article discusses mandatory submission, ClinicalTrials.gov also accepts voluntary submissions).[ref-17] Importantly, this trumps any excuse pertaining to journal/editorial rejection.

On the note of ClinicalTrials.gov and the length to which I discuss it (comment 1), I do so because ClinicalTrials.gov was ushered in with great fanfare, and yet it has unequivocally failed and was doomed at the outset to be a partial success at best in its original (and current) form. Such assertions are admittedly bold, so the space I devote to dissecting this issue was seen as necessary, and truthfully, the space I devote to this topic is ultimately relatively limited given the evidence on the matter. However, the failure of ClinicalsTrials.gov to rectify the issues of publication bias and inadequate research transparency does not preclude it from being used in the aforementioned manner; moreover, with some important logistical changes and better oversight and enforcement (all of which I address in my article), ClinicalTrials.gov could actually be a remarkable tool in the fight to end publication bias and inadequate research transparency.

Further questioning the role the journals/editors might play, Ms. Bowles also questions the willingness of a top-tier journal to publish non-significant results, asking when a top-tier journal last published a non-significant result (comment 1, the response to which will also further address comment 3). However, there are several recent examples of studies with “null,” “negative,” or “non-significant” results even within the so-called “Big 5,” including (in alphabetic order) Annals of Internal Medicine,[ref-18] The BMJ,[ref-19]^,[ref-20]Journal of the American Medical Association,[ref-21]^,[ref-22] The Lancet,[ref-23]^,[ref-24] and New England Journal of Medicine.[ref-25]^,[ref-26]

Again, considering all of this, although the evidence cannot entirely rule out journal/editorial bias, the totality of the best-available evidence nevertheless suggests the majority of the problem of publication bias rests more with authors/researchers, not with journals/editors. However, as I discuss clearly in my article, journals/editors are not blameless; indeed, a large area of failure has been the disappointing failure to uphold the ICMJE proclamation.

Ms. Bowles also comments on the seeming focus on clinical trials (comment 1), expresses a desire for further coverage of observational studies (comment 2), and wonders whether observational studies might suffer an even worse fate than trials (comment 2). I would like more research on observational studies as well, but as Song and colleagues note in their 2010 systematic review:

“Empirical studies on publication and related biases have focused mainly on certain areas of research such as clinical trials of health-care interventions. There is only very limited evidence on publication bias in many other research fields including basic research and observational studies.”[ref-2]^(p36)

Although the data are limited, there are some investigations into the matter, although many actually include a mixture of study types.[ref-2]^,[ref-11]^,[ref-27]^-[ref-30] When considering Ms. Bowles’ curiosity as to whether observational studies suffer even more from publication bias, the limitations in the available data ultimately preclude one from making any definitive statements, and the data we do have are not congruent. In a subgroup analysis, Easterbrook and colleagues found observational and laboratory-based experimental studies considered together may be more susceptible to publication bias than randomized controlled trials.[ref-29] However, Dickersin and colleagues did not detect any difference in clinical trials versus other studies,[ref-11] and Stern and Simes found the risk of publication bias was even stronger for the subgroup of clinical trials included in their study compared to all the studies considered together.[ref-30] Returning to the 2010 systematic review, Song and colleagues note the results on publication bias were materially indifferent when analyses were restricted to clinical trials versus including all studies.[ref-2]

Although observational data are certainly useful, it seems at least possible that the focus on clinical trials is due to clinical trials representing a higher, more reliable level of evidence when making inferences about an intervention’s effects, a sentiment echoed by Dwan and colleagues in their 2013 systematic review.[ref-27] It also seems at least feasible that this is why efforts to improve the situation have focused on clinical trials. This certainly is not to say observational data are unimportant; indeed, we should strive to gain access to all data, not just data from trials. Perhaps the hopeful success of efforts currently underway for trials can be used as a framework for other study types; in the meantime, however, operating within the existing frameworks and efforts and within the confines of reality, it seems reasonable and important to continue with the efforts focused on clinical trials, understanding that no sensible person would consider gaining proper access to untainted clinical trial data as being the end of this plight if there is still evidence of issues with other study types. The end goal is, as Silverman opined, to be able to truly “narrow the area of uncertainty” as much as possible to build “working truths … that support our everyday actions at the bedside.”[ref-31]^(p165) As long as issues with publication bias and inadequate research transparency persist, this goal remains elusive, and we must therefore continue to fight until these issues are a sordid tale in our past.

Lastly, Ms. Bowles comments that I do not mention the impact that publication bias has on future research, specifically that missing research may result in research being unknowingly repeated, thereby potentially leading to wasting time and money (comment 4). However, I actually do hint at this indirectly in the last sentence of my introduction (“These issues distort the evidence base from which medical providers and researchers should be operating, and therefore put patients at risk of direct or indirect harm.”) and the second-to-last sentence of my conclusion (“Indeed, publication bias and inadequate research transparency represent a gaping wound in the body of evidence from which researchers operate and medical providers make decisions regarding care.”). I also more directly state the effects on research time and funding earlier in my conclusion:

“Continuing to tolerate these problems is not only an ethical blemish of the worst kind, but is also: wholly disrespectful to those who participate in clinical trials with the belief their participation will help improve care; a cause of inefficient use of research time and funding; and perhaps most importantly, a threat to providers’ ability to provide their patients with the best evidence-based care possible.”

Still, I thank Ms. Bowles for her comment here, as I do think I could have been more explicit when discussing how these issues can lead to inefficient use of research time and funding, and I think being more explicit would have made for an even stronger overall statement of the multifaceted deleterious effects these issues have.

I wholeheartedly agree with Ms. Bowles that the NIH should heavily penalize authors who fail to publish properly, and I honestly think this penalization should be extended beyond the NIH. Indeed, it seems monitoring and penalties (that are actually enforced) will be a necessary step in improving these issues. This speaks to part of the reason why I discuss in such detail how the government has failed to enforce Section 801 of the Food and Drug Administration Amendments Act of 2007, how the changes proposed in 2014 (including changes specific to trials receiving funding from the NIH) still fail to address enforcement and do not seem to offer much hope of true improvement, how the ICMJE proclamation has also not been upheld, and how current efforts underway seek to improve the state of affairs.

In summary: The available evidence, while not able to exculpate journals/editors entirely, still suggests authors/researchers are more to blame for publication bias than journals/editors; journals/editors seem willing to publish “negative”/non-significant results; we need more investigation into how these issues affect observational studies, but it is reasonable and important to continue with efforts focused on clinical trials for the time being (understanding that the fight will continue until publication bias and inadequate research transparency are truly a thing of the past); and these issues can lead to substantial wasting of time and funding due to unawareness of previously-conducted studies that were not properly disseminated. As I state in my article, previous measures have failed to rectify the issues of publication bias and inadequate research transparency (though these measures could prove meaningful with logistical changes, better oversight, and actual enforcement), current movements have gained vigor, and one hopes they will not fall short like their predecessors. Again, we must continue to pursue these matters until true resolution occurs; anything less than that is unacceptable and disgraceful.

[References]

[[1|type=journal|authors=Emerson/GB;Warme/WJ;Wolf/FM;Heckman/JD;Brand/RA;Leopold/SS|title=Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial|source=Arch Intern Med|year=2010|vol=170|issue=21|fpage=1934|lpage=1939|doi=10.1001/archinternmed.2010.406|pmid=21098355]][[2|type=journal|authors=Song/F;Parekh/S;Hooper/L;Loke/YK;Ryder/J;Sutton/AJ;Hing/C;Kwok/CS;Pang/C;Harvey/I|title=Dissemination and publication of research findings: an updated review of related biases|source=Health Technol Assess|year=2010|vol=14|issue=8|doi=10.3310/hta14080|pmid=20181324]][[3|type=journal|authors=Olson/CM;Rennie/D;Cook/D;Dickersin/K;Flanagin/A;Hogan/JW;Zhu/Q;Reiling/J;Pace/B|title=Publication bias in editorial decision making|source=JAMA|year=2002|vol=287|issue=21|fpage=2825|lpage=2828|pmid=12038924]]
[[4|type=journal|authors=Lee/KP;Boyd/EA;Holroyd-Leduc/JM;Bacchetti/P;Bero/LA|title=Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals|source=Med J Aust|year=2006|vol=184|issue=12|fpage=621|lpage=626|pmid=16803442]][[5|type=journal|authors=Lynch/JR;Cunningham/MRA;Warme/WJ;Schaad/DC;Wolf/FM;Leopold/SS|title=Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not|source=J Bone Joint Surg Am|year=2007|vol=89|issue=5|fpage=1010|lpage=1018|pmid=17473138]][[6|type=journal|authors=Okike/K;Kocher/MS;Mehlman/CT;Heckman/JD;Bhandari/M|title=Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume)|source=J Bone Joint Surg Am|year=2008|vol=90|issue=3|fpage=595|lpage=601|doi=10.2106/JBJS.G.00279|pmid=18310710]]
[[7|type=journal|authors=van Lent/M;Overbeke/J;Out/HJ|title=Role of editorial and peer review processes in publication bias: analysis of drug trials submitted to eight medical journals|source=PLoS One|year=2014|vol=9|issue=8|fpage=e104846|doi=10.1371/journal.pone.0104846|pmid=25118182|pmcid=4130599]]
[[8|type=journal|authors=Ioannidis/JPA|title=Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials|source=JAMA|year=1998|vol=279|issue=4|fpage=281|lpage=286|pmid=9450711]][[9|type=journal|authors=Dickersin/K;Olson/CM;Rennie/D;Cook/D;Flanagin/A;Zhu/Q;Reiling/J;Pace/B|title=Association between time interval to publication and statistical significance|source=JAMA|year=2002|vol=287|issue=21|fpage=2829|lpage=2831|pmid=12038925]][[10|type=journal|authors=Dickersin/K;Chan/S;Chalmers/TC;Sacks/HS;Smith/H Jr|title=Publication bias and clinical trials|source=Control Clin Trials|year=1987|vol=8|issue=4|fpage=343|lpage=353|pmid=3442991]]
[[11|type=journal|authors=Dickersin/K;Min/Y-I;Meinert/CL|title=Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards|source=JAMA|year=1992|vol=267|issue=3|fpage=374|lpage=378|pmid=1727960]]
[[12|type=journal|authors=Dickersin/K;Min/YI|title=Publication bias: the problem that won’t go away|source=Ann N Y Acad Sci|year=1993|vol=703|fpage=135|lpage=148|pmid=8192291]][[13|type=journal|authors=Weber/EJ;Callaham/ML;Wears/RL;Barton/C;Young/G|title=Unpublished research from a medical specialty meeting: why investigators fail to publish|source=JAMA|year=1998|vol=280|issue=3|fpage=257|lpage=259|pmid=9676674]][[14|type=journal|authors=Timmer/A;Hilsden/RJ;Cole/J;Hailey/D;Sutherland/LR|title=Publication bias in gastroenterological research - a retrospective cohort study based on abstracts submitted to a scientific meeting|source=BMC Med Res Methodol|year=2002|vol=2|fpage=7|pmid=11978183|pmcid=115863]]
[[15|type=journal|authors=Krzyzanowska/MK;Pintilie/M;Tannock/IF|title=Factors associated with failure to publish large randomized trials presented at an oncology meeting|source=JAMA|year=2003|vol=290|issue=4|fpage=495|lpage=501|pmid=12876092]]
[[16|type=journal|authors=Scherer/RW;Langenberg/P|title=Full publication of results initially presented in abstracts|source=Cochrane Database Syst Rev|year=2007|vol=18|issue=2|fpage=MR000005|pmid=17443628]]
[[17|authors=US National Insitutes of Health|title=Frequently Asked Questions|source=ClinicalTrials.gov|fpage=Last reviewed February 2015|lpage=Accessed October 11, 2015|url=https://www.clinicaltrials.gov/ct2/manage-recs/faq]]
[[18|type=journal|authors=Schleussner/E;Kamin/G;Seliger/G;Rogenhofer/N;Ebner/S;Toth/B;Schenk/M;Henes/M;Bohlmann/MK;Fischer/T;Brosteanu/O;Bauersachs/R;Petroff/D;ETHIG II group|title=Low-molecular-weight heparin for women with unexplained recurrent pregnancy loss: a multicenter trial with a minimization randomization scheme|source=Ann Intern Med|year=2015|vol=162|issue=9|fpage=601|lpage=609|doi=10.7326/M14-2062|pmid=25938990]]
[[19|type=journal|authors=Bolland/MJ;Leung/W;Tai/V;Bastin/S;Gamble/GD;Grey/A;Reid/IR|title=Calcium intake and risk of fracture: systematic review|source=BMJ|year=2015|vol=351|fpage=h4580|doi=10.1136/bmj.h4580|pmid=26420387]]
[[20|title=Efficacy and safety of paracetamol for spinal pain and osteoarthritis: systematic review and meta-analysis of randomised placebo controlled trials|authors=Machado/GC;Maher/CG;Ferreira/PH;Pinheiro/MB;Lin/CW;Day/RO;McLachlan/AJ;Ferreira/ML|source=BMJ|vol=350|issue=h1225|year=2015|type=journal|doi=10.1136/bmj.h1225|pmid=25828856|pmcid=PMC4381278|url=http://www.bmj.com/content/350/bmj.h1225.long]]
[[21|title=Effect of omega-3 fatty acids, lutein/zeaxanthin, or other nutrient supplementation on cognitive function: the AREDS2 randomized clinical trial|authors=Chew/EY;Clemons/TE;Agrón/E;Launer/LJ;Grodstein/F;Bernstein/PS;Age-Related Eye Disease Study 2 (AREDS2) Research Group|source=JAMA|vol=314|issue=8|year=2015|fpage=791|lpage=801|type=journal|doi=10.1001/jama.2015.9677|pmid=26305649|url=http://jama.jamanetwork.com/article.aspx?articleid=2429713]]
[[22|title=Surgical vs nonsurgical treatment of adults with displaced fractures of the proximal humerus: the PROFHER randomized clinical trial|authors=Rangan/A;Handoll/H;Brealey/S;Jefferson/L;Keding/A;Martin/BC;Goodchild/L;Chuang/LH;Hewitt/C;Torgerson/D;PROFHER Trial Collaborators|source=JAMA|vol=313|issue=10|year=2015|fpage=1037|lpage=1047|type=journal|doi=10.1001/jama.2015.1629|pmid=25756440|pmcid=25756440|url=http://jama.jamanetwork.com/article.aspx?articleid=2190987]]
[[23|title=Methylprednisolone in patients undergoing cardiopulmonary bypass (SIRS): a randomised, double-blind, placebo-controlled trial|authors=Whitlock/RP;Devereaux/PJ;Teoh/KH;Lamy/A;Vincent/J;Pogue/J;Paparella/D;Sessler/DI;Karthikeyan/G;Villar/JC;Zuo/Y;Avezum/Á;Quantz/M;Tagarakis/GI;Shah/PJ;Abbasi/SH;Zheng/H;Pettit/S;Chrolavicius/S;Yusuf/S|source=Lancet|vol=386|issue=10000|year=2015|fpage=1243|lpage=1253|type=journal|doi= 10.1016/S0140-6736(15)00273-1|url=http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2815%2900273-1/abstract]]
[[24|title=Medical expulsive therapy in adults with ureteric colic: a multicentre, randomised, placebo-controlled trial|authors=Pickard/R;Starr/K;MacLennan/G;Lam/T;Thomas/R;Burr/J;McPherson/G;McDonald/A;Anson/K;N'Dow/J;Burgess/N;Clark/T;Kilonzo/M;Gillies/K;Shearer/K;Boachie/C;Cameron/S;Norrie/J;McClinton/S|source=Lancet|vol=386|issue=9991|year=2015|fpage=341|lpage=349|type=journal|doi= 10.1016/S0140-6736(15)60933-3|pmid=25998582|url=http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2815%2960933-3/abstract]]
[[25|title=Spironolactone for heart failure with preserved ejection fraction|authors=Pitt/B;Pfeffer/MA;Assmann/SF;Boineau/R;Anand/IS;Claggett/B;Clausell/N;Desai;AS;Diaz/R;Fleg/JL;Gordeev/I;Harty/B;Heitner/JF;Kenwood/CT;Lewis/EF;O'Meara/E;Probstfield/JL;Shaburishvili/T;Shah/SJ;Solomon/SD;Sweitzer/NK;Yang/S;McKinlay/SM;TOPCAT Investigators|source=N Engl J Med|vol=370|issue=15|year=2014|fpage=1383|lpage=1392|type=journal|doi=10.1056/NEJMoa1313731|pmid=24716680|url=http://www.nejm.org/doi/full/10.1056/NEJMoa1313731]]
[[26|title=Cyclosporine before PCI in Patients with Acute Myocardial Infarction|authors=Cung/TT;Morel/O;Cayla/G;Rioufol/G;Garcia-Dorado/D;Angoulvant/D;Bonnefoy-Cudraz/E;Guérin/P;Elbaz/M;Delarche/N;Coste/P;Vanzetto/G;Metge/M;Aupetit/J-F;Jouve/B;Motreff/P;Tron/C;Labeque/J-N;Steg/PG;Cottin/Y; Range/G;Clerc/J;Claeys/MJ;Coussement/P;Prunier/F;Moulin/F;Roth/M;Belle/L;Dubois/P;Barragan/P;Gilard/M;Piot/C;Colin/P;De Poli/F;Morice/M-C;Ider/O;Dubois-Randé/J-L;Unterseeh/T;Le Breton/H;Béard/T;Blanchard/D;Grollier/G;Malquarti/V;Staat/P;Sudre/A;Elmer/E;Hansson/MJ;Bergerot/C;Boussaha/I;Jossan/C;Derumeaux/G;Mewton/N;Michel/O|source=N Engl J Med|vol=373|year=2015|fpage=1021|lpage=1031|type=journal|doi=10.1056/NEJMoa1505489|url=http://www.nejm.org/doi/full/10.1056/NEJMoa1505489]]
[[27|title=Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review|authors=Dwan/K;Altman/DG;Arnaiz/JA;Bloom/J;Chan/AW;Cronin/E;Decullier/E;Easterbrook/PJ;Von Elm/E;Gamble/C;Ghersi/D;Ioannidis/JP;Simes/J;Williamson/PR|source=PLoS One|vol=3|issue=8|year=2008|fpage=e3081|type=journal|doi=10.1371/journal.pone.0003081|pmid=18769481|pmcid=PMC2518111|url=http://www.ncbi.nlm.nih.gov/pubmed/18769481]]
[[28|type=journal|authors=Kavvoura/FK;Liberopoulos/G;Ioannidis/JP|title=Selection in reported epidemiological risks: an empirical assessment|source=PLoS Med|year=2007|vol=4|issue=3|fpage=e79|pmid=17341129 |pmcid=1808481]]
[[29|type=journal|authors=Easterbrook/PJ;Berlin/JA;Gopalan/R;Matthews/DR|title=Publication bias in clinical research|source=Lancet|year=1991|vol=337|issue=8746|fpage=867|lpage=872|pmid=1672966]]
[[30|title=Publication bias: evidence of delayed publication in a cohort study of clinical research projects|authors=Stern/JM;Simes/RJ|source=BMJ|vol=315|issue=7109|year=1997|fpage=640|lpage=645|type=journal|pmid=9310565|url=http://www.bmj.com/content/315/7109/640]]
[[31|title=Where's the Evidence?: Debates in Modern Medicine|authors=Silverman/WA|vol=Oxford University Press|issue=Oxford|year=1999|type=book|url=http://ukcatalogue.oup.com/product/9780192630889.do#]]
I am glad Ms. Bowles appreciates the gravity of these issues, and I am grateful for her overall positive review of my article. My article was not designed as a primary research article, and therefore there were no primary scientific data to report. Still, although this does mean one cannot judge scientific quality in terms of methodological appraisal for a primary research article (or a meta-analysis, etc.), I would nevertheless posit my article can still be judged as a positive contribution to the biomedical sciences, as one can ultimately think of science in two broad, interrelated senses: Firstly, science is ultimately a way of thinking about and approaching phenomena that involves asking answerable questions and then systematically and impartially pursuing and using the totality of the best-available evidence to reach an informed and reasonable stance, a stance that should remain appropriately malleable as new evidence accumulates and gets incorporated into the current body of knowledge on a subject; secondly, science is a collaborative endeavor to achieve such goals, to collegially seek out such working truths as best we can.

This collaborative aspect speaks to my sincere appreciation of Ms. Bowles’ review and comments regarding my article. Below, I respond to her comments at length (with the comment[s] to which I am responding indicated in the responses).

Ms. Bowles comments that much attention is paid to ClinicalTrials.gov, but that biases from journals may have been overlooked, and that she believes this is where much of the bias lies (comment 1); she is also concerned that the willingness of certain journals to publish “null results” might have been overlooked, and that this could be part of the solution (comment 3). To be sure, however, no such oversights occurred; although I originally decided to omit an in-depth discussion of such matters in my article for the sake of length and for reasons that will become clear as I respond to these comments, I will gladly expound here, and the available evidence does offer some reassurance and illumination, though it does not support Ms. Bowles’ comments.

Although bias in the peer-review process has been demonstrated,¹ the best-available evidence overall suggests the majority of the problem actually rests with researchers/authors failing to publish or otherwise make available the results of their research, not journals/editors preferentially publishing only studies with “positive”/significant findings.²^-¹⁴ Indeed, this is discussed in the 2010 systematic review I cite in my article, which pooled the four studies to date that had investigated this matter.² Song and colleagues found a pooled odds ratio of acceptance of 1.06 for manuscripts with “positive”/significant findings versus manuscripts with “negative”/non-significant findings (95% confidence interval [CI], 0.80 to 1.39), which they rightly conclude as corroborating the notion that acceptance of a submitted manuscript was not significantly associated with whether the paper had “positive”/significant or “negative”/non-significant findings. The four studies included in the pooled analysis included two studies investigating general medical journals (JAMA in one study,³ and The BMJ, The Lancet, and Annals of Internal Medicine in the other⁴) and two studies investigating the American volume of the Journal of Bone and Joint Surgery.⁵^,⁶ All these studies individually arrived at the same result as well: once submitted to a journal for consideration, manuscripts with “positive”/significant findings are no more likely to be published than manuscripts with “negative”/non-significant findings.

One of the studies⁵ – which investigated only manuscripts on hip or knee arthroplasty submitted to the American volume of the Journal of Bone and Joint Surgery – has a potentially suggestive title, and it drew attention to the significant difference in the average quality of “positive”/significant manuscripts versus “negative”/non-significant manuscripts when quality was assessed by a score derived by the authors from standard criteria used to assess the quality of a piece of evidence (termed the Sackett score by the authors). On average, “negative”/non-significant manuscripts had higher Sackett scores than “positive”/significant manuscripts. When considering the mean level of evidence scale officially required by the journal during manuscript preparation and submission, however, there was no significant difference between “positive”/significant manuscripts and “negative”/non-significant manuscripts. There was no difference in acceptance based on study outcome, and intriguingly, there was no difference in acceptance based on study quality as assessed by either the mean Sackett score or the mean level of evidence scale officially used and required by the journal. This finding causes one to ponder, as one would rightly expect higher-quality manuscripts to be published with greater frequency than lower-quality manuscripts. They discuss the finding of no difference in quality between accepted versus rejected manuscripts thusly:

“Potential explanations for this finding include the possibility that the overall quality of submissions to The Journal is so high that quality differences among them are too subtle to be discerned with use of standard evidence-based-medicine scoring approaches or, more likely, that reviewers make their decisions on the basis of other criteria, such as timeliness, economic impact, or perceived relevance, in addition to those recommended in standard evidence-based-medicine texts.”⁵^(p1016)

However, these postulated explanations still leave one with at least some degree of lingering uncertainty regarding what to think about this finding and how to reconcile this finding with the other assessments and discussions of quality in this study. The authors also acknowledge that assessing manuscript quality and any relationship with acceptance versus rejection as being “outside the scope of [the] primary study end points as defined by [their] a priori hypotheses” and that “until or unless this finding is validated by others it should not be accorded great weight.” ⁵^(p1016) (Indeed, investigating study quality was not a part of either of the two clearly-stated primary research hypotheses.)

Furthermore, a large weakness of this study is its lack of attempt to control for potentially influential or confounding variables. Considering all of this, one should be cautious about making any strong conclusions about any findings of manuscript quality or relationships pertaining to manuscript quality and publication from this study, and even setting all this aside, one must also remember this study looked at submissions to one specialty journal and included only studies on hip and knee arthroplasty. Even the subsequent study investigating submissions to the American volume of the Journal of Bone and Joint Surgery noted that, due to the other study’s limitations, “it remains unclear whether publication bias exists in the orthopaedic journal editorial review process.”⁶^(p596) The subsequent study thus set its primary research intent as seeking to determine whether manuscripts with “positive”/significant findings were more likely to be published after controlling for various quality measures, level of evidence, and subspecialty field.⁶ In this study and the two studies that investigated general medical journals³^,⁴ – all of which investigated quality measures and utilized multivariate analyses to control for potentially influential or confounding variables – no difference was found in publication rates for submitted manuscripts with “positive”/significant results versus submitted manuscripts with “negative”/non-significant results. These studies also found that manuscripts of higher quality or level of evidence were significantly more likely to be published.³^,⁴^,⁶

Recently, a retrospective review of manuscripts from several journals (The BMJ, Annals of the Rheumatic Diseases, British Journal of Ophthalmology, Gut, Heart, Thorax, Diabetologia, and Journal of Hepatology) was published in PLoS ONE,⁷ and although retrospective studies are often viewed as having potential weaknesses compared to prospective studies, the retrospective design offers a unique benefit in this case: the elimination of any Hawthorne effect. Such an effect might occur in the aforementioned prospective studies on this issue, since editors and reviewers might, whether consciously or unconsciously, behave somewhat differently if they are aware their actions are being studied. This retrospective study also found no difference in acceptance for publication for submitted manuscripts with “positive”/significant findings compared to submitted manuscripts with “negative”/non-significant findings, this study considered quality indicators as well, and the finding of no difference was present in both the univariate and multivariate analyses.

However, what if merely looking at publication in a binary manner misses a level of nuance that might still have an impact? What if journals/editors are less enthusiastic about publishing “negative”/non-significant results compared to “positive”/significant results, and therefore, whether unconsciously or consciously, do not pursue the steps to publication as diligently, leading to significant delays in publication? Ioannidis found that, among a cohort of 66 completed multicenter trials focused on human immunodeficiency virus, the 45 trials that had been submitted for publication were published slightly more quickly after submission if they had “positive”/significant results (median of 0.8 years versus 1.1 years; P = 0.04); however, the ultimate clinical significance of an approximate 3.6 month delay is unclear.⁸ However, in another study investigating a cohort of 133 trials submitted to JAMA, there was no difference in time from submission to publication for “negative”/non-significant results compared to “positive”/significant results (median of 7.6 months versus 7.8 months, respectively; P = 0.44).⁹

Of course, a potential limitation in these studies is the inherent inability to account for manuscripts/studies that are not even written up and submitted to journals. The theoretical question of what these results would look like if non-submitted manuscripts were submitted is certainly an interesting question, even if only for the sake of refining our understanding of this matter. In the absence of such, however, the aforementioned studies remain our best evidence on the matter, and it would be entirely unfair to hold any speculative counterfactual against the journals/editors.

In summary, when considering what is known about publication bias and what this evidence shows, while the evidence may not allow us to confidently exculpate journals/editors entirely, it surely corroborates the notion that authors/researchers are more culpable than journals/editors.

Further insight can be gained from asking the researchers themselves why they failed to write up and submit a study.²^,¹⁰^-¹⁶ Here too, we see the majority of the problem seems to rest not with the journals/editors, but with the authors/researchers, with the most common reasons for not writing up and submitting a study tending to be lack of time or lack of interest. Both reasons are wholly unacceptable. Being rejected by a journal actually tends to be reported by only a minority, but this too is an unacceptable excuse; manuscript rejection is largely to be expected at some point in one’s publishing endeavors (regardless of the manuscript’s findings/content), and allowing this to be a deterrent is a fault of the researchers/authors, not the journals/editors. Due to cognitive biases, authors/researchers may be more likely to assign fault or biased motives to the journal/editors (and thus may also view things in a more emotionally charged manner) when a manuscript with “negative”/non-significant results is rejected compared to when a manuscript with “positive”/significant results is rejected; however, this is merely a speculative suggestion, albeit one that would seem to fit within what we know about human behavior. In any case, one must remember authors/researchers are specifically called to adhere to the World Medical Association’s Declaration of Helsinki in making the results of their research on humans publicly available (see the second sentence of principle 36 in Figure 2 in my article). Furthermore, at least for clinical trials, ClinicalTrials.gov could serve as a repository for study methods and results even if the authors/researchers do not wish to seek traditional publication or get rejected and decide not to pursue publication elsewhere (although my article discusses mandatory submission, ClinicalTrials.gov also accepts voluntary submissions).¹⁷ Importantly, this trumps any excuse pertaining to journal/editorial rejection.

On the note of ClinicalTrials.gov and the length to which I discuss it (comment 1), I do so because ClinicalTrials.gov was ushered in with great fanfare, and yet it has unequivocally failed and was doomed at the outset to be a partial success at best in its original (and current) form. Such assertions are admittedly bold, so the space I devote to dissecting this issue was seen as necessary, and truthfully, the space I devote to this topic is ultimately relatively limited given the evidence on the matter. However, the failure of ClinicalsTrials.gov to rectify the issues of publication bias and inadequate research transparency does not preclude it from being used in the aforementioned manner; moreover, with some important logistical changes and better oversight and enforcement (all of which I address in my article), ClinicalTrials.gov could actually be a remarkable tool in the fight to end publication bias and inadequate research transparency.

Further questioning the role the journals/editors might play, Ms. Bowles also questions the willingness of a top-tier journal to publish non-significant results, asking when a top-tier journal last published a non-significant result (comment 1, the response to which will also further address comment 3). However, there are several recent examples of studies with “null,” “negative,” or “non-significant” results even within the so-called “Big 5,” including (in alphabetic order) Annals of Internal Medicine,¹⁸ The BMJ,¹⁹^,²⁰Journal of the American Medical Association,²¹^,²² The Lancet,²³^,²⁴ and New England Journal of Medicine.²⁵^,²⁶

Again, considering all of this, although the evidence cannot entirely rule out journal/editorial bias, the totality of the best-available evidence nevertheless suggests the majority of the problem of publication bias rests more with authors/researchers, not with journals/editors. However, as I discuss clearly in my article, journals/editors are not blameless; indeed, a large area of failure has been the disappointing failure to uphold the ICMJE proclamation.

Ms. Bowles also comments on the seeming focus on clinical trials (comment 1), expresses a desire for further coverage of observational studies (comment 2), and wonders whether observational studies might suffer an even worse fate than trials (comment 2). I would like more research on observational studies as well, but as Song and colleagues note in their 2010 systematic review:

“Empirical studies on publication and related biases have focused mainly on certain areas of research such as clinical trials of health-care interventions. There is only very limited evidence on publication bias in many other research fields including basic research and observational studies.”²^(p36)

Although the data are limited, there are some investigations into the matter, although many actually include a mixture of study types.²^,¹¹^,²⁷^-³⁰ When considering Ms. Bowles’ curiosity as to whether observational studies suffer even more from publication bias, the limitations in the available data ultimately preclude one from making any definitive statements, and the data we do have are not congruent. In a subgroup analysis, Easterbrook and colleagues found observational and laboratory-based experimental studies considered together may be more susceptible to publication bias than randomized controlled trials.²⁹ However, Dickersin and colleagues did not detect any difference in clinical trials versus other studies,¹¹ and Stern and Simes found the risk of publication bias was even stronger for the subgroup of clinical trials included in their study compared to all the studies considered together.³⁰ Returning to the 2010 systematic review, Song and colleagues note the results on publication bias were materially indifferent when analyses were restricted to clinical trials versus including all studies.²

Although observational data are certainly useful, it seems at least possible that the focus on clinical trials is due to clinical trials representing a higher, more reliable level of evidence when making inferences about an intervention’s effects, a sentiment echoed by Dwan and colleagues in their 2013 systematic review.²⁷ It also seems at least feasible that this is why efforts to improve the situation have focused on clinical trials. This certainly is not to say observational data are unimportant; indeed, we should strive to gain access to all data, not just data from trials. Perhaps the hopeful success of efforts currently underway for trials can be used as a framework for other study types; in the meantime, however, operating within the existing frameworks and efforts and within the confines of reality, it seems reasonable and important to continue with the efforts focused on clinical trials, understanding that no sensible person would consider gaining proper access to untainted clinical trial data as being the end of this plight if there is still evidence of issues with other study types. The end goal is, as Silverman opined, to be able to truly “narrow the area of uncertainty” as much as possible to build “working truths … that support our everyday actions at the bedside.”³¹^(p165) As long as issues with publication bias and inadequate research transparency persist, this goal remains elusive, and we must therefore continue to fight until these issues are a sordid tale in our past.

Lastly, Ms. Bowles comments that I do not mention the impact that publication bias has on future research, specifically that missing research may result in research being unknowingly repeated, thereby potentially leading to wasting time and money (comment 4). However, I actually do hint at this indirectly in the last sentence of my introduction (“These issues distort the evidence base from which medical providers and researchers should be operating, and therefore put patients at risk of direct or indirect harm.”) and the second-to-last sentence of my conclusion (“Indeed, publication bias and inadequate research transparency represent a gaping wound in the body of evidence from which researchers operate and medical providers make decisions regarding care.”). I also more directly state the effects on research time and funding earlier in my conclusion:

“Continuing to tolerate these problems is not only an ethical blemish of the worst kind, but is also: wholly disrespectful to those who participate in clinical trials with the belief their participation will help improve care; a cause of inefficient use of research time and funding; and perhaps most importantly, a threat to providers’ ability to provide their patients with the best evidence-based care possible.”

Still, I thank Ms. Bowles for her comment here, as I do think I could have been more explicit when discussing how these issues can lead to inefficient use of research time and funding, and I think being more explicit would have made for an even stronger overall statement of the multifaceted deleterious effects these issues have.

I wholeheartedly agree with Ms. Bowles that the NIH should heavily penalize authors who fail to publish properly, and I honestly think this penalization should be extended beyond the NIH. Indeed, it seems monitoring and penalties (that are actually enforced) will be a necessary step in improving these issues. This speaks to part of the reason why I discuss in such detail how the government has failed to enforce Section 801 of the Food and Drug Administration Amendments Act of 2007, how the changes proposed in 2014 (including changes specific to trials receiving funding from the NIH) still fail to address enforcement and do not seem to offer much hope of true improvement, how the ICMJE proclamation has also not been upheld, and how current efforts underway seek to improve the state of affairs.

In summary: The available evidence, while not able to exculpate journals/editors entirely, still suggests authors/researchers are more to blame for publication bias than journals/editors; journals/editors seem willing to publish “negative”/non-significant results; we need more investigation into how these issues affect observational studies, but it is reasonable and important to continue with efforts focused on clinical trials for the time being (understanding that the fight will continue until publication bias and inadequate research transparency are truly a thing of the past); and these issues can lead to substantial wasting of time and funding due to unawareness of previously-conducted studies that were not properly disseminated. As I state in my article, previous measures have failed to rectify the issues of publication bias and inadequate research transparency (though these measures could prove meaningful with logistical changes, better oversight, and actual enforcement), current movements have gained vigor, and one hopes they will not fall short like their predecessors. Again, we must continue to pursue these matters until true resolution occurs; anything less than that is unacceptable and disgraceful.

References
1. Emerson GB, Warme WJ, Wolf FM, Heckman JD, et al.: Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Arch Intern Med. 2010; 170 (21): 1934-1939 PubMed Abstract | Publisher Full Text
2. Song F, Parekh S, Hooper L, Loke YK, et al.: Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010; 14 (8). PubMed Abstract | Publisher Full Text
3. Olson CM, Rennie D, Cook D, Dickersin K, et al.: Publication bias in editorial decision making. JAMA. 2002; 287 (21): 2825-2828 PubMed Abstract
4. Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, et al.: Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006; 184 (12): 621-626 PubMed Abstract
5. Lynch JR, Cunningham MRA, Warme WJ, Schaad DC, et al.: Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. J Bone Joint Surg Am. 2007; 89 (5): 1010-1018 PubMed Abstract
6. Okike K, Kocher MS, Mehlman CT, Heckman JD, et al.: Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008; 90 (3): 595-601 PubMed Abstract | Publisher Full Text
7. van Lent M, Overbeke J, Out HJ: Role of editorial and peer review processes in publication bias: analysis of drug trials submitted to eight medical journals. PLoS One. 2014; 9 (8): e104846 PubMed Abstract | Free Full Text | Publisher Full Text
8. Ioannidis JPA: Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA. 1998; 279 (4): 281-286 PubMed Abstract
9. Dickersin K, Olson CM, Rennie D, Cook D, et al.: Association between time interval to publication and statistical significance. JAMA. 2002; 287 (21): 2829-2831 PubMed Abstract
10. Dickersin K, Chan S, Chalmers TC, Sacks HS, et al.: Publication bias and clinical trials. Control Clin Trials. 1987; 8 (4): 343-353 PubMed Abstract
11. Dickersin K, Min Y-I, Meinert CL: Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 1992; 267 (3): 374-378 PubMed Abstract
12. Dickersin K, Min YI: Publication bias: the problem that won’t go away. Ann N Y Acad Sci. 1993; 703: 135-148 PubMed Abstract
13. Weber EJ, Callaham ML, Wears RL, Barton C, et al.: Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA. 1998; 280 (3): 257-259 PubMed Abstract
14. Timmer A, Hilsden RJ, Cole J, Hailey D, et al.: Publication bias in gastroenterological research - a retrospective cohort study based on abstracts submitted to a scientific meeting. BMC Med Res Methodol. 2002; 2: 7 PubMed Abstract | Free Full Text
15. Krzyzanowska MK, Pintilie M, Tannock IF: Factors associated with failure to publish large randomized trials presented at an oncology meeting. JAMA. 2003; 290 (4): 495-501 PubMed Abstract
16. Scherer RW, Langenberg P: Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007; 18 (2): MR000005 PubMed Abstract
17. US National Insitutes of Health: Frequently Asked Questions. ClinicalTrials.gov. Last reviewed February 2015-Accessed October 11, 2015 Reference Source
18. Schleussner E, Kamin G, Seliger G, Rogenhofer N, et al.: Low-molecular-weight heparin for women with unexplained recurrent pregnancy loss: a multicenter trial with a minimization randomization scheme. Ann Intern Med. 2015; 162 (9): 601-609 PubMed Abstract | Publisher Full Text
19. Bolland MJ, Leung W, Tai V, Bastin S, et al.: Calcium intake and risk of fracture: systematic review. BMJ. 2015; 351: h4580 PubMed Abstract | Publisher Full Text
20. Machado GC, Maher CG, Ferreira PH, Pinheiro MB, et al.: Efficacy and safety of paracetamol for spinal pain and osteoarthritis: systematic review and meta-analysis of randomised placebo controlled trials. BMJ. 2015; 350 (h1225). PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
21. Chew EY, Clemons TE, Agrón E, Launer LJ, et al.: Effect of omega-3 fatty acids, lutein/zeaxanthin, or other nutrient supplementation on cognitive function: the AREDS2 randomized clinical trial. JAMA. 2015; 314 (8): 791-801 PubMed Abstract | Publisher Full Text | Reference Source
22. Rangan A, Handoll H, Brealey S, Jefferson L, et al.: Surgical vs nonsurgical treatment of adults with displaced fractures of the proximal humerus: the PROFHER randomized clinical trial. JAMA. 2015; 313 (10): 1037-1047 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
23. Whitlock RP, Devereaux PJ, Teoh KH, Lamy A, et al.: Methylprednisolone in patients undergoing cardiopulmonary bypass (SIRS): a randomised, double-blind, placebo-controlled trial. Lancet. 2015; 386 (10000): 1243-1253 Publisher Full Text | Reference Source
24. Pickard R, Starr K, MacLennan G, Lam T, et al.: Medical expulsive therapy in adults with ureteric colic: a multicentre, randomised, placebo-controlled trial. Lancet. 2015; 386 (9991): 341-349 PubMed Abstract | Publisher Full Text | Reference Source
25. Pitt B, Pfeffer MA, Assmann SF, Boineau R, et al.: Spironolactone for heart failure with preserved ejection fraction. N Engl J Med. 2014; 370 (15): 1383-1392 PubMed Abstract | Publisher Full Text | Reference Source
26. Cung TT, Morel O, Cayla G, Rioufol G, et al.: Cyclosporine before PCI in Patients with Acute Myocardial Infarction. N Engl J Med. 2015; 373: 1021-1031 Publisher Full Text | Reference Source
27. Dwan K, Altman DG, Arnaiz JA, Bloom J, et al.: Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review. PLoS One. 2008; 3 (8): e3081 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
28. Kavvoura FK, Liberopoulos G, Ioannidis JP: Selection in reported epidemiological risks: an empirical assessment. PLoS Med. 2007; 4 (3): e79 PubMed Abstract | Free Full Text
29. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991; 337 (8746): 867-872 PubMed Abstract
30. Stern JM, Simes RJ: Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997; 315 (7109): 640-645 PubMed Abstract | Reference Source
31. Silverman WA: Where's the Evidence?: Debates in Modern Medicine. 1999; Oxford University Press (Oxford). Reference Source
Competing Interests: Please see my original competing interests statement, which remains in effect and unchanged since publishing my original article. Close
Report a concern

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 16 Sep 2015

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1
Version 1 16 Sep 15	read

Erin Aiello Bowles, Group Health Research Institute, Seattle, USA

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

37 Views

02 Oct 2015 | for Version 1

Erin Aiello Bowles, Group Health Research Institute, Seattle, WA, USA

37 Views Cite this report Responses(1)

Approved With Reservations

The article spends much of the paper focusing on clinical trials. Nearly an entire page is devoted to ClinicalTrials.gov. While this is interesting, it is not the only cause of publication bias. I would cut down some of this information and provide information on additional causes. For example, what about biases from journals? When was the last time a top-tier journal published a non-significant result, even if the result was important. I believe this is where much of the bias lies. Perhaps breaking this into 2 sections would help – one section on bias as a result of the author failing to report results properly (or at all) and one section on bias as a result of journals failing to publish articles that aren’t highly statistically significant. The latter is a difficulty that many authors face that needs more attention if policies will ever change, and may even be part of the reason authors fail to properly report results.
I would also encourage the author to expand on publication bias in observational studies. This is mentioned briefly on page 3 and may be an even bigger problem than publication bias in clinical trials.
The author doesn’t mention that some journals (CEBP being one) have specific calls for “null results” to reduce publication bias. Additional calls and journal policies on actively publishing null results may be one solution to this problem.
The author also doesn’t mention the impact that publication bias has on future research. When a study isn’t published, no one knows. No one knows what the results were or whether the trial worked. So the study may be funded again in a different form and repeated only to find the same null result. This is a complete waste of time and our tax payer dollars, particularly for government-funded studies. The NIH should really crack down on the lack of publications from grants and prohibit funding of future similar studies if the initial results are never published.

Competing Interests

No competing interests were disclosed.

Respond to this report

Responses (1)

Author Response

13 Oct 2015

Martin Mayer, Department of Physician Assistant Studies, College of Allied Health Sciences, East Carolina University, Greenville, 27858-4353, USA

I am glad Ms. Bowles appreciates the gravity of these issues, and I am grateful for her overall positive review of my article. My article was not designed as a primary research article, and therefore there were no primary scientific data to report. Still, although this does mean one cannot judge scientific quality in terms of methodological appraisal for a primary research article (or a meta-analysis, etc.), I would nevertheless posit my article can still be judged as a positive contribution to the biomedical sciences, as one can ultimately think of science in two broad, interrelated senses: Firstly, science is ultimately a way of thinking about and approaching phenomena that involves asking answerable questions and then systematically and impartially pursuing and using the totality of the best-available evidence to reach an informed and reasonable stance, a stance that should remain appropriately malleable as new evidence accumulates and gets incorporated into the current body of knowledge on a subject; secondly, science is a collaborative endeavor to achieve such goals, to collegially seek out such working truths as best we can.

This collaborative aspect speaks to my sincere appreciation of Ms. Bowles’ review and comments regarding my article. Below, I respond to her comments at length (with the comment[s] to which I am responding indicated in the responses).

Ms. Bowles comments that much attention is paid to ClinicalTrials.gov, but that biases from journals may have been overlooked, and that she believes this is where much of the bias lies (comment 1); she is also concerned that the willingness of certain journals to publish “null results” might have been overlooked, and that this could be part of the solution (comment 3). To be sure, however, no such oversights occurred; although I originally decided to omit an in-depth discussion of such matters in my article for the sake of length and for reasons that will become clear as I respond to these comments, I will gladly expound here, and the available evidence does offer some reassurance and illumination, though it does not support Ms. Bowles’ comments.

Although bias in the peer-review process has been demonstrated,¹ the best-available evidence overall suggests the majority of the problem actually rests with researchers/authors failing to publish or otherwise make available the results of their research, not journals/editors preferentially publishing only studies with “positive”/significant findings.²^-¹⁴ Indeed, this is discussed in the 2010 systematic review I cite in my article, which pooled the four studies to date that had investigated this matter.² Song and colleagues found a pooled odds ratio of acceptance of 1.06 for manuscripts with “positive”/significant findings versus manuscripts with “negative”/non-significant findings (95% confidence interval [CI], 0.80 to 1.39), which they rightly conclude as corroborating the notion that acceptance of a submitted manuscript was not significantly associated with whether the paper had “positive”/significant or “negative”/non-significant findings. The four studies included in the pooled analysis included two studies investigating general medical journals (JAMA in one study,³ and The BMJ, The Lancet, and Annals of Internal Medicine in the other⁴) and two studies investigating the American volume of the Journal of Bone and Joint Surgery.⁵^,⁶ All these studies individually arrived at the same result as well: once submitted to a journal for consideration, manuscripts with “positive”/significant findings are no more likely to be published than manuscripts with “negative”/non-significant findings.

One of the studies⁵ – which investigated only manuscripts on hip or knee arthroplasty submitted to the American volume of the Journal of Bone and Joint Surgery – has a potentially suggestive title, and it drew attention to the significant difference in the average quality of “positive”/significant manuscripts versus “negative”/non-significant manuscripts when quality was assessed by a score derived by the authors from standard criteria used to assess the quality of a piece of evidence (termed the Sackett score by the authors). On average, “negative”/non-significant manuscripts had higher Sackett scores than “positive”/significant manuscripts. When considering the mean level of evidence scale officially required by the journal during manuscript preparation and submission, however, there was no significant difference between “positive”/significant manuscripts and “negative”/non-significant manuscripts. There was no difference in acceptance based on study outcome, and intriguingly, there was no difference in acceptance based on study quality as assessed by either the mean Sackett score or the mean level of evidence scale officially used and required by the journal. This finding causes one to ponder, as one would rightly expect higher-quality manuscripts to be published with greater frequency than lower-quality manuscripts. They discuss the finding of no difference in quality between accepted versus rejected manuscripts thusly:

“Potential explanations for this finding include the possibility that the overall quality of submissions to The Journal is so high that quality differences among them are too subtle to be discerned with use of standard evidence-based-medicine scoring approaches or, more likely, that reviewers make their decisions on the basis of other criteria, such as timeliness, economic impact, or perceived relevance, in addition to those recommended in standard evidence-based-medicine texts.”⁵^(p1016)

However, these postulated explanations still leave one with at least some degree of lingering uncertainty regarding what to think about this finding and how to reconcile this finding with the other assessments and discussions of quality in this study. The authors also acknowledge that assessing manuscript quality and any relationship with acceptance versus rejection as being “outside the scope of [the] primary study end points as defined by [their] a priori hypotheses” and that “until or unless this finding is validated by others it should not be accorded great weight.” ⁵^(p1016) (Indeed, investigating study quality was not a part of either of the two clearly-stated primary research hypotheses.)

Furthermore, a large weakness of this study is its lack of attempt to control for potentially influential or confounding variables. Considering all of this, one should be cautious about making any strong conclusions about any findings of manuscript quality or relationships pertaining to manuscript quality and publication from this study, and even setting all this aside, one must also remember this study looked at submissions to one specialty journal and included only studies on hip and knee arthroplasty. Even the subsequent study investigating submissions to the American volume of the Journal of Bone and Joint Surgery noted that, due to the other study’s limitations, “it remains unclear whether publication bias exists in the orthopaedic journal editorial review process.”⁶^(p596) The subsequent study thus set its primary research intent as seeking to determine whether manuscripts with “positive”/significant findings were more likely to be published after controlling for various quality measures, level of evidence, and subspecialty field.⁶ In this study and the two studies that investigated general medical journals³^,⁴ – all of which investigated quality measures and utilized multivariate analyses to control for potentially influential or confounding variables – no difference was found in publication rates for submitted manuscripts with “positive”/significant results versus submitted manuscripts with “negative”/non-significant results. These studies also found that manuscripts of higher quality or level of evidence were significantly more likely to be published.³^,⁴^,⁶

Recently, a retrospective review of manuscripts from several journals (The BMJ, Annals of the Rheumatic Diseases, British Journal of Ophthalmology, Gut, Heart, Thorax, Diabetologia, and Journal of Hepatology) was published in PLoS ONE,⁷ and although retrospective studies are often viewed as having potential weaknesses compared to prospective studies, the retrospective design offers a unique benefit in this case: the elimination of any Hawthorne effect. Such an effect might occur in the aforementioned prospective studies on this issue, since editors and reviewers might, whether consciously or unconsciously, behave somewhat differently if they are aware their actions are being studied. This retrospective study also found no difference in acceptance for publication for submitted manuscripts with “positive”/significant findings compared to submitted manuscripts with “negative”/non-significant findings, this study considered quality indicators as well, and the finding of no difference was present in both the univariate and multivariate analyses.

However, what if merely looking at publication in a binary manner misses a level of nuance that might still have an impact? What if journals/editors are less enthusiastic about publishing “negative”/non-significant results compared to “positive”/significant results, and therefore, whether unconsciously or consciously, do not pursue the steps to publication as diligently, leading to significant delays in publication? Ioannidis found that, among a cohort of 66 completed multicenter trials focused on human immunodeficiency virus, the 45 trials that had been submitted for publication were published slightly more quickly after submission if they had “positive”/significant results (median of 0.8 years versus 1.1 years; P = 0.04); however, the ultimate clinical significance of an approximate 3.6 month delay is unclear.⁸ However, in another study investigating a cohort of 133 trials submitted to JAMA, there was no difference in time from submission to publication for “negative”/non-significant results compared to “positive”/significant results (median of 7.6 months versus 7.8 months, respectively; P = 0.44).⁹

Of course, a potential limitation in these studies is the inherent inability to account for manuscripts/studies that are not even written up and submitted to journals. The theoretical question of what these results would look like if non-submitted manuscripts were submitted is certainly an interesting question, even if only for the sake of refining our understanding of this matter. In the absence of such, however, the aforementioned studies remain our best evidence on the matter, and it would be entirely unfair to hold any speculative counterfactual against the journals/editors.

In summary, when considering what is known about publication bias and what this evidence shows, while the evidence may not allow us to confidently exculpate journals/editors entirely, it surely corroborates the notion that authors/researchers are more culpable than journals/editors.

Further insight can be gained from asking the researchers themselves why they failed to write up and submit a study.²^,¹⁰^-¹⁶ Here too, we see the majority of the problem seems to rest not with the journals/editors, but with the authors/researchers, with the most common reasons for not writing up and submitting a study tending to be lack of time or lack of interest. Both reasons are wholly unacceptable. Being rejected by a journal actually tends to be reported by only a minority, but this too is an unacceptable excuse; manuscript rejection is largely to be expected at some point in one’s publishing endeavors (regardless of the manuscript’s findings/content), and allowing this to be a deterrent is a fault of the researchers/authors, not the journals/editors. Due to cognitive biases, authors/researchers may be more likely to assign fault or biased motives to the journal/editors (and thus may also view things in a more emotionally charged manner) when a manuscript with “negative”/non-significant results is rejected compared to when a manuscript with “positive”/significant results is rejected; however, this is merely a speculative suggestion, albeit one that would seem to fit within what we know about human behavior. In any case, one must remember authors/researchers are specifically called to adhere to the World Medical Association’s Declaration of Helsinki in making the results of their research on humans publicly available (see the second sentence of principle 36 in Figure 2 in my article). Furthermore, at least for clinical trials, ClinicalTrials.gov could serve as a repository for study methods and results even if the authors/researchers do not wish to seek traditional publication or get rejected and decide not to pursue publication elsewhere (although my article discusses mandatory submission, ClinicalTrials.gov also accepts voluntary submissions).¹⁷ Importantly, this trumps any excuse pertaining to journal/editorial rejection.

On the note of ClinicalTrials.gov and the length to which I discuss it (comment 1), I do so because ClinicalTrials.gov was ushered in with great fanfare, and yet it has unequivocally failed and was doomed at the outset to be a partial success at best in its original (and current) form. Such assertions are admittedly bold, so the space I devote to dissecting this issue was seen as necessary, and truthfully, the space I devote to this topic is ultimately relatively limited given the evidence on the matter. However, the failure of ClinicalsTrials.gov to rectify the issues of publication bias and inadequate research transparency does not preclude it from being used in the aforementioned manner; moreover, with some important logistical changes and better oversight and enforcement (all of which I address in my article), ClinicalTrials.gov could actually be a remarkable tool in the fight to end publication bias and inadequate research transparency.

Further questioning the role the journals/editors might play, Ms. Bowles also questions the willingness of a top-tier journal to publish non-significant results, asking when a top-tier journal last published a non-significant result (comment 1, the response to which will also further address comment 3). However, there are several recent examples of studies with “null,” “negative,” or “non-significant” results even within the so-called “Big 5,” including (in alphabetic order) Annals of Internal Medicine,¹⁸ The BMJ,¹⁹^,²⁰Journal of the American Medical Association,²¹^,²² The Lancet,²³^,²⁴ and New England Journal of Medicine.²⁵^,²⁶

Again, considering all of this, although the evidence cannot entirely rule out journal/editorial bias, the totality of the best-available evidence nevertheless suggests the majority of the problem of publication bias rests more with authors/researchers, not with journals/editors. However, as I discuss clearly in my article, journals/editors are not blameless; indeed, a large area of failure has been the disappointing failure to uphold the ICMJE proclamation.

Ms. Bowles also comments on the seeming focus on clinical trials (comment 1), expresses a desire for further coverage of observational studies (comment 2), and wonders whether observational studies might suffer an even worse fate than trials (comment 2). I would like more research on observational studies as well, but as Song and colleagues note in their 2010 systematic review:

“Empirical studies on publication and related biases have focused mainly on certain areas of research such as clinical trials of health-care interventions. There is only very limited evidence on publication bias in many other research fields including basic research and observational studies.”²^(p36)

Although the data are limited, there are some investigations into the matter, although many actually include a mixture of study types.²^,¹¹^,²⁷^-³⁰ When considering Ms. Bowles’ curiosity as to whether observational studies suffer even more from publication bias, the limitations in the available data ultimately preclude one from making any definitive statements, and the data we do have are not congruent. In a subgroup analysis, Easterbrook and colleagues found observational and laboratory-based experimental studies considered together may be more susceptible to publication bias than randomized controlled trials.²⁹ However, Dickersin and colleagues did not detect any difference in clinical trials versus other studies,¹¹ and Stern and Simes found the risk of publication bias was even stronger for the subgroup of clinical trials included in their study compared to all the studies considered together.³⁰ Returning to the 2010 systematic review, Song and colleagues note the results on publication bias were materially indifferent when analyses were restricted to clinical trials versus including all studies.²

Although observational data are certainly useful, it seems at least possible that the focus on clinical trials is due to clinical trials representing a higher, more reliable level of evidence when making inferences about an intervention’s effects, a sentiment echoed by Dwan and colleagues in their 2013 systematic review.²⁷ It also seems at least feasible that this is why efforts to improve the situation have focused on clinical trials. This certainly is not to say observational data are unimportant; indeed, we should strive to gain access to all data, not just data from trials. Perhaps the hopeful success of efforts currently underway for trials can be used as a framework for other study types; in the meantime, however, operating within the existing frameworks and efforts and within the confines of reality, it seems reasonable and important to continue with the efforts focused on clinical trials, understanding that no sensible person would consider gaining proper access to untainted clinical trial data as being the end of this plight if there is still evidence of issues with other study types. The end goal is, as Silverman opined, to be able to truly “narrow the area of uncertainty” as much as possible to build “working truths … that support our everyday actions at the bedside.”³¹^(p165) As long as issues with publication bias and inadequate research transparency persist, this goal remains elusive, and we must therefore continue to fight until these issues are a sordid tale in our past.

Lastly, Ms. Bowles comments that I do not mention the impact that publication bias has on future research, specifically that missing research may result in research being unknowingly repeated, thereby potentially leading to wasting time and money (comment 4). However, I actually do hint at this indirectly in the last sentence of my introduction (“These issues distort the evidence base from which medical providers and researchers should be operating, and therefore put patients at risk of direct or indirect harm.”) and the second-to-last sentence of my conclusion (“Indeed, publication bias and inadequate research transparency represent a gaping wound in the body of evidence from which researchers operate and medical providers make decisions regarding care.”). I also more directly state the effects on research time and funding earlier in my conclusion:

“Continuing to tolerate these problems is not only an ethical blemish of the worst kind, but is also: wholly disrespectful to those who participate in clinical trials with the belief their participation will help improve care; a cause of inefficient use of research time and funding; and perhaps most importantly, a threat to providers’ ability to provide their patients with the best evidence-based care possible.”

Still, I thank Ms. Bowles for her comment here, as I do think I could have been more explicit when discussing how these issues can lead to inefficient use of research time and funding, and I think being more explicit would have made for an even stronger overall statement of the multifaceted deleterious effects these issues have.

I wholeheartedly agree with Ms. Bowles that the NIH should heavily penalize authors who fail to publish properly, and I honestly think this penalization should be extended beyond the NIH. Indeed, it seems monitoring and penalties (that are actually enforced) will be a necessary step in improving these issues. This speaks to part of the reason why I discuss in such detail how the government has failed to enforce Section 801 of the Food and Drug Administration Amendments Act of 2007, how the changes proposed in 2014 (including changes specific to trials receiving funding from the NIH) still fail to address enforcement and do not seem to offer much hope of true improvement, how the ICMJE proclamation has also not been upheld, and how current efforts underway seek to improve the state of affairs.

In summary: The available evidence, while not able to exculpate journals/editors entirely, still suggests authors/researchers are more to blame for publication bias than journals/editors; journals/editors seem willing to publish “negative”/non-significant results; we need more investigation into how these issues affect observational studies, but it is reasonable and important to continue with efforts focused on clinical trials for the time being (understanding that the fight will continue until publication bias and inadequate research transparency are truly a thing of the past); and these issues can lead to substantial wasting of time and funding due to unawareness of previously-conducted studies that were not properly disseminated. As I state in my article, previous measures have failed to rectify the issues of publication bias and inadequate research transparency (though these measures could prove meaningful with logistical changes, better oversight, and actual enforcement), current movements have gained vigor, and one hopes they will not fall short like their predecessors. Again, we must continue to pursue these matters until true resolution occurs; anything less than that is unacceptable and disgraceful.

References

1. Emerson GB, Warme WJ, Wolf FM, Heckman JD, et al.: Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Arch Intern Med. 2010; 170 (21): 1934-1939 PubMed Abstract | Publisher Full Text
2. Song F, Parekh S, Hooper L, Loke YK, et al.: Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010; 14 (8). PubMed Abstract | Publisher Full Text
3. Olson CM, Rennie D, Cook D, Dickersin K, et al.: Publication bias in editorial decision making. JAMA. 2002; 287 (21): 2825-2828 PubMed Abstract
4. Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, et al.: Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006; 184 (12): 621-626 PubMed Abstract
5. Lynch JR, Cunningham MRA, Warme WJ, Schaad DC, et al.: Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. J Bone Joint Surg Am. 2007; 89 (5): 1010-1018 PubMed Abstract
6. Okike K, Kocher MS, Mehlman CT, Heckman JD, et al.: Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008; 90 (3): 595-601 PubMed Abstract | Publisher Full Text
7. van Lent M, Overbeke J, Out HJ: Role of editorial and peer review processes in publication bias: analysis of drug trials submitted to eight medical journals. PLoS One. 2014; 9 (8): e104846 PubMed Abstract | Free Full Text | Publisher Full Text
8. Ioannidis JPA: Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA. 1998; 279 (4): 281-286 PubMed Abstract
9. Dickersin K, Olson CM, Rennie D, Cook D, et al.: Association between time interval to publication and statistical significance. JAMA. 2002; 287 (21): 2829-2831 PubMed Abstract
10. Dickersin K, Chan S, Chalmers TC, Sacks HS, et al.: Publication bias and clinical trials. Control Clin Trials. 1987; 8 (4): 343-353 PubMed Abstract
11. Dickersin K, Min Y-I, Meinert CL: Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 1992; 267 (3): 374-378 PubMed Abstract
12. Dickersin K, Min YI: Publication bias: the problem that won’t go away. Ann N Y Acad Sci. 1993; 703: 135-148 PubMed Abstract
13. Weber EJ, Callaham ML, Wears RL, Barton C, et al.: Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA. 1998; 280 (3): 257-259 PubMed Abstract
14. Timmer A, Hilsden RJ, Cole J, Hailey D, et al.: Publication bias in gastroenterological research - a retrospective cohort study based on abstracts submitted to a scientific meeting. BMC Med Res Methodol. 2002; 2: 7 PubMed Abstract | Free Full Text
15. Krzyzanowska MK, Pintilie M, Tannock IF: Factors associated with failure to publish large randomized trials presented at an oncology meeting. JAMA. 2003; 290 (4): 495-501 PubMed Abstract
16. Scherer RW, Langenberg P: Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007; 18 (2): MR000005 PubMed Abstract
17. US National Insitutes of Health: Frequently Asked Questions. ClinicalTrials.gov. Last reviewed February 2015-Accessed October 11, 2015 Reference Source
18. Schleussner E, Kamin G, Seliger G, Rogenhofer N, et al.: Low-molecular-weight heparin for women with unexplained recurrent pregnancy loss: a multicenter trial with a minimization randomization scheme. Ann Intern Med. 2015; 162 (9): 601-609 PubMed Abstract | Publisher Full Text
19. Bolland MJ, Leung W, Tai V, Bastin S, et al.: Calcium intake and risk of fracture: systematic review. BMJ. 2015; 351: h4580 PubMed Abstract | Publisher Full Text
20. Machado GC, Maher CG, Ferreira PH, Pinheiro MB, et al.: Efficacy and safety of paracetamol for spinal pain and osteoarthritis: systematic review and meta-analysis of randomised placebo controlled trials. BMJ. 2015; 350 (h1225). PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
21. Chew EY, Clemons TE, Agrón E, Launer LJ, et al.: Effect of omega-3 fatty acids, lutein/zeaxanthin, or other nutrient supplementation on cognitive function: the AREDS2 randomized clinical trial. JAMA. 2015; 314 (8): 791-801 PubMed Abstract | Publisher Full Text | Reference Source
22. Rangan A, Handoll H, Brealey S, Jefferson L, et al.: Surgical vs nonsurgical treatment of adults with displaced fractures of the proximal humerus: the PROFHER randomized clinical trial. JAMA. 2015; 313 (10): 1037-1047 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
23. Whitlock RP, Devereaux PJ, Teoh KH, Lamy A, et al.: Methylprednisolone in patients undergoing cardiopulmonary bypass (SIRS): a randomised, double-blind, placebo-controlled trial. Lancet. 2015; 386 (10000): 1243-1253 Publisher Full Text | Reference Source
24. Pickard R, Starr K, MacLennan G, Lam T, et al.: Medical expulsive therapy in adults with ureteric colic: a multicentre, randomised, placebo-controlled trial. Lancet. 2015; 386 (9991): 341-349 PubMed Abstract | Publisher Full Text | Reference Source
25. Pitt B, Pfeffer MA, Assmann SF, Boineau R, et al.: Spironolactone for heart failure with preserved ejection fraction. N Engl J Med. 2014; 370 (15): 1383-1392 PubMed Abstract | Publisher Full Text | Reference Source
26. Cung TT, Morel O, Cayla G, Rioufol G, et al.: Cyclosporine before PCI in Patients with Acute Myocardial Infarction. N Engl J Med. 2015; 373: 1021-1031 Publisher Full Text | Reference Source
27. Dwan K, Altman DG, Arnaiz JA, Bloom J, et al.: Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review. PLoS One. 2008; 3 (8): e3081 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
28. Kavvoura FK, Liberopoulos G, Ioannidis JP: Selection in reported epidemiological risks: an empirical assessment. PLoS Med. 2007; 4 (3): e79 PubMed Abstract | Free Full Text
29. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991; 337 (8746): 867-872 PubMed Abstract
30. Stern JM, Simes RJ: Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997; 315 (7109): 640-645 PubMed Abstract | Reference Source
31. Silverman WA: Where's the Evidence?: Debates in Modern Medicine. 1999; Oxford University Press (Oxford). Reference Source

View more View less

Competing Interests

Please see my original competing interests statement, which remains in effect and unchanged since publishing my original article.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Editorial. The reporting of unsuccessful cases. Boston Med Surg J. 1909; 161: 263–264. Reference Source

[2] 2. Dickersin K: Publication bias: recognizing the problem, understanding its origins and scope, and preventing harm. In: Rothstein HR, Sutton AJ and Borenstein M, eds. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. West Sussex, England: John Wiley & Sons; 2005; 11–33. Publisher Full Text

[3] 3. Ferriar J: Medical Histories and Reflections. London: Cadell and Davies, 1792; 1. Reference Source

[4] 4. Petticrew M: Diagoras of Melos (500 BC): An early analyst of publication bias. Lancet. 1998; 352(9139): 1558. PubMed Abstract | Publisher Full Text

[5] 5. Sterling T: Publication decisions and their possible effects on inferences drawn from tests of significance -- Or vice versa. J Am Stat Assoc. 1959; 54(285): 30–34. Publisher Full Text

[6] 6. Sterling TD, Rosenbaum WL, Weinkam JJ: Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. Am Stat. 1995; 49(1): 108–112. Publisher Full Text

[7] 7. Dwan K, Gamble C, Williamson PR, et al.: Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One. 2013; 8(7): e66844. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Song F, Parekh S, Hooper L, et al.: Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010; 14(8): iii, ix–xi, 1–193. PubMed Abstract | Publisher Full Text

[9] 9. Hopewell S, Loudon K, Clarke MJ, et al.: Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009; (1): MR000006. Accession # 00075320-100000000-05507. PubMed Abstract | Publisher Full Text

[10] 10. Wieseler B, Wolfram N, McGauran N, et al.: Completeness of reporting of patient-relevant clinical trial outcomes: comparison of unpublished clinical study reports with publicly available data. PLoS Med. 2013; 10(10): e1001526. PubMed Abstract | Publisher Full Text | Free Full Text

[11] 11. Dwan K, Altman DG, Cresswell L, et al.: Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev. 2011; (1): MR000031. PubMed Abstract | Publisher Full Text

[12] 12. Chan AW, Hróbjartsson A, Haahr MT, et al.: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004; 291(20): 2457–2465. PubMed Abstract | Publisher Full Text

[13] 13. Mayer M: Peering into the “rabbit hole” of publication bias and inadequate research transparency: Adding tangibility to the abstract [v1; ref status: awaiting peer review, http://f1000r.es/5pp]. F1000Res. 2015; 4: 609. Publisher Full Text

[14] 14. De Angelis C, Drazen JM, Frizelle FA, et al.: Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004; 351(12): 1250–1251. PubMed Abstract | Publisher Full Text

[15] 15. De Angelis CD, Drazen JM, Frizelle FA, et al.: Is this clinical trial fully registered?--A statement from the International Committee of Medical Journal Editors. N Engl J Med. 2005; 352(23): 2436–2438. PubMed Abstract | Publisher Full Text

[16] 16. Mathieu S, Boutron I, Moher D, et al.: Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009; 302(9): 977–84. PubMed Abstract | Publisher Full Text

[17] 17. Jones CW, Platts-Mills TF: Quality of registration for clinical trials published in emergency medicine journals. Ann Emerg Med. 2012; 60(4): 458–464.e1. PubMed Abstract | Publisher Full Text

[18] 18. Killeen S, Sourallous P, Hunter IA, et al.: Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals. Ann Surg. 2014; 259(1): 193–196. PubMed Abstract | Publisher Full Text

[19] 19. Scott A, Rucklidge JJ, Mulder RT: Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. PLoS One. 2015; 10(8): e0133718. PubMed Abstract | Publisher Full Text | Free Full Text

[20] 20. The Food and Drug Administration. Title VIII, Section 801, Food and Drug Administration Amendments Act of 2007. 2007. [Pages 82 (http://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf#page=82) through 100 (http://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf#page=100) of the Food and Drug Administration Amendments Act of 2007 (http://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf)] Reference Source

[21] 21. U.S. National Institutes of Health. FDAAA 801 requirements. 2014. Last reviewed December 2014. Accessed/link verified December 19, 2014. Reference Source

[22] 22. Prayle AP, Hurley MN, Smyth AR: Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ. 2012; 344: d7373. PubMed Abstract | Publisher Full Text

[23] 23. Huser V, Cimino JJ: Linking ClinicalTrials.gov and PubMed to track results of interventional human clinical trials. PLoS One. 2013; 8(7): e68409. PubMed Abstract | Publisher Full Text | Free Full Text

[24] 24. Jones CW, Handler L, Crowell KE, et al.: Non-publication of large randomized clinical trials: cross sectional analysis. BMJ. 2013; 347: f6104. PubMed Abstract | Publisher Full Text | Free Full Text

[25] 25. Law MR, Kawasumi Y, Morgan SG: Despite law, fewer than one in eight completed studies of drugs and biologics are reported on time on ClinicalTrials.gov. Health Aff (Millwood). 2011; 30(12): 2338–45. PubMed Abstract | Publisher Full Text

[26] 26. Kirillova O: Results and outcome reporting in ClinicalTrials.gov, what makes it happen? PLoS One. 2012; 7(6): e37847. PubMed Abstract | Publisher Full Text | Free Full Text

[27] 27. Ross JS, Mulvey GK, Hines EM, et al.: Trial publication after registration in ClinicalTrials.Gov: a cross-sectional analysis. PLoS Med. 2009; 6(9): e1000144. PubMed Abstract | Publisher Full Text | Free Full Text

[28] 28. Anderson ML, Chiswell K, Peterson ED, et al.: Compliance with results reporting at ClinicalTrials.gov. N Engl J Med. 2015; 372(11): 1031–1039. PubMed Abstract | Publisher Full Text | Free Full Text

[29] 29. Mathieu S, Chan AW, Ravaud P: Use of trial register information during the peer review process. PLoS One. 2013; 8(4): e59910. PubMed Abstract | Publisher Full Text | Free Full Text

[30] 30. U.S. National Institutes of Health, Department of Health and Human Services. 42 CFR Part 11 [Docket Number NIH– 2011– 0003], RIN 0925–AA52, Clinical Trials Registration and Results Submission, Action: Notice of Proposed Rulemaking. Federal Register. 2014; 79(225): Accessed link/verified January 1, 2015. Reference Source

[31] 31. U.S. National Institutes of Health. HHS and NIH take steps to enhance transparency of clinical trial results. Published November 19, 2014; Reviewed November 21, 2014. Accessed/link verified November 22, 2014. Reference Source

[32] 32. U.S. National Institutes of Health. Summary of HHS/NIH proposals to enhance transparency of clinical trial results. Reviewed November 19, 2014; Accessed/link verified November 22, 2014. Reference Source

[33] 33. European Medicines Agency. European Medicines Agency policy on publication of clinical data for medicinal products for human use. 2014. Accessed/link verified January 1, 2015. Reference Source

[34] 34. Adams B: The pioneers of transparency. BMJ. 2015; 350: g7717. PubMed Abstract | Publisher Full Text

[35] 35. Doshi P, Dickersin K, Healy D, et al.: Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ. 2013; 346: f2865. PubMed Abstract | Publisher Full Text | Free Full Text

[36] 36. Treasure T, Monson K, Fiorentino F, et al.: The CEA Second-Look Trial: a randomised controlled trial of carcinoembryonic antigen prompted reoperation for recurrent colorectal cancer. BMJ Open. 2014; 4(5): e004385. PubMed Abstract | Publisher Full Text | Free Full Text

[37] 37. Doshi P: Putting GlaxoSmithKline to the test over paroxetine. BMJ. 2013; 347: f6754. PubMed Abstract | Publisher Full Text

[38] 38. Strom BL, Buyse M, Hughes J, et al.: Data sharing, year 1--access to data from industry-sponsored clinical trials. N Engl J Med. 2014; 371(22): 2052–4. PubMed Abstract | Publisher Full Text

[39] 39. Zarin DA: Participant-level data and the new frontier in trial transparency. N Engl J Med. 2013; 369(5): 468–469. PubMed Abstract | Publisher Full Text

[40] 40. Jureidini JN, Nardo JM: Inadequacy of remote desktop interface for independent reanalysis of data from drug trials. BMJ. 2014; 349: g4353. PubMed Abstract | Publisher Full Text

[41] 41. Lo B, Coetzee T, Demets D, et al.: Committee on Strategies for Responsible Sharing of Clinical Trial Data, Board on Health Sciences Policy, Institute of Medicine. Sharing clinical trial data: Maximizing benefits, minimizing risk. National Academies Press, 2015. ISBN 978-0-309-31629-3. Accessed/link verified January 17, 2015. Reference Source

[42] 42. Moorthy VS, Karam G, Vannice KS, et al.: Rationale for WHO's new position calling for prompt reporting and public disclosure of interventional clinical trial results. PLoS Med. 2015; 12(4): e1001819. PubMed Abstract | Publisher Full Text | Free Full Text

[43] 43. Hawkes N: Lifetime achievement award 2014: Sir Iain Chalmers. BMJ. 2014; 348: g2921. PubMed Abstract | Publisher Full Text

[44] 44. Chalmers I: Underreporting research is scientific misconduct. JAMA. 1990; 263(10): 1405–1408. PubMed Abstract | Publisher Full Text

[45] 45. Chalmers I: Proposal to outlaw the term “negative trial”. BMJ (Clin Res Ed). 1985; 290(6473): 1002. Publisher Full Text | Free Full Text

[46] 46. World Medical Association. Declaration of Helsinki: ethical principles for medical research involving human subjects. 2013. Accessed/link verified August 6, 2014. Reference Source

A call to arms to help heal medicine’s greatest ailment - Publication bias and inadequate research transparency

Abstract

Keywords

Introduction

Figure 1. An editorial published in 1909 in The Boston Medical and Surgical Journal.1,2

Well-intended but ineffective remedial attempts

Table 1. Compliance estimates from Anderson and colleagues’ analysis of compliance with reporting results for highly likely applicable clinical trials based on funding and time period.28

Potential remedies currently being tested

Conclusion

Figure 2. Key principles from the World Medical Association’s Declaration of Helsinki addressing the need for transparency and proper dissemination of research involving human subjects.

Competing interests

Grant information

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated

Figure 1. An editorial published in 1909 in The Boston Medical and Surgical Journal.^1,2

Table 1. Compliance estimates from Anderson and colleagues’ analysis of compliance with reporting results for highly likely applicable clinical trials based on funding and time period.²⁸