F1000ResearchF1000Research2046-1402F1000 Research LimitedLondon, UK10.12688/f1000research.2-1.v1CorrespondenceArticlesPublishing & Peer ReviewBiased under-reporting of research reflects biased under-submission more than biased editorial rejection
[version 1; peer review: 3 approved]
ChalmersIaina1DickersinKay2
James Lind Initiative, Summertown Pavilion, Middle Way, Oxford, OX2 7LG, UK
Center for Clinical Trials, Johns Hopkins Bloomberg School of Public Health, 615 N. Wolfe St. Rm E6152, Baltimore, Maryland, 21205, USAichalmers@jameslindlibrary.org
The authors both contributed to the text submitted.
Competing interests: No competing interests were disclosed.
Stephen Senn challenges Ben Goldacre’s assertion in ‘
Bad Pharma’ that biased editorial acceptance of reports with ‘positive’ findings is not a cause of biased under-reporting of research. We agree with Senn that biased editorial decisions may contribute to reporting bias, but Senn ignores the evidence that biased decisions by researchers to submit reports for possible publication are the main causes of the problem.
peer reviewThe author(s) declared that no grants were involved in supporting this work.
Stephen Senn challenges Ben Goldacre’s assertion in ‘
Bad Pharma’
1 that biased editorial acceptance of reports with ‘positive’ findings is not a cause of biased under-reporting of research, and concludes that "the prospects for disentangling cause and effect when it comes to publication bias are not great"
2. Senn apparently overlooks the studies – including controlled experiments - which have investigated reporting biases. These are summarised in an article
3 from which the following is an excerpt:
“
Who is responsible for biased reporting of clinical research?
Reporting bias can be due to researchers and sponsors failing to submit study findings for publication, or due to journal editors and others rejecting reports for publication. Numerous surveys of investigators have left little doubt that almost all failure to publish is due to the failure of investigators to submit reports for publication4,
5,
with only a small proportion of studies remaining unpublished because of rejection by journals6,
although positive-outcome bias has been demonstrated among peer reviewers7.
Qualitative studies of editorial discussion indicate that a study’s scientific rigour is the area of greatest concern8.
Researchers report that the reason they do not write up and submit reports of their research for publication is usually because they are "not interested" in the results ("editorial rejection by journals" is only rarely given as a cause of failure to publish). Even those investigators who have initially published their results as (conference) abstracts are less likely to submit their findings for full publication unless the results are ‘significant’9.
Investigations of biased reporting of research began with surveys of journal articles, which revealed improbably high proportions of published studies showing statistically significant differences10–
14.
Subsequent surveys of authors and peer reviewers showed that research that had yielded ‘negative’ results was less likely than other research to be submitted or recommended for publication15–
18.
These findings have been reinforced by the results of experimental studies, which showed that studies with no reported statistically significant differences were less likely to be accepted for publication7,
19–
21".
Senn’s use of the term ‘publication bias’ in his commentary suggests that he is restricting it to editorial bias whereas, as indicated above, the origins of reporting bias are largely due to researchers’ decisions not to submit, not editorial decisions not to accept. The analyses of observational data cited by Ben Goldacre in his book ‘
Bad Pharma’
1 do not detect editorial bias, but neither do they support a confident conclusion that no editorial bias exists. However, we believe Goldacre is correct to castigate researchers and research sponsors as being more culpable than editors in betraying their responsibility to the patients who have participated in trials.
The controlled experiments suggest that it is the results of studies, not their quality, that predisposes them to editorial bias. Senn believes that any editorial bias that exists can be ‘very plausibly explained’ by preferential publication of ‘positive’ studies, and that it "seems plausible that higher quality studies are more likely to lead to a positive result". Unless he is using the word ‘positive’ to mean something other than ‘a beneficial effect’, however, Senn appears to be overlooking substantial evidence challenging the plausibility of his belief (see, for example, reference
22). Given the estimated likelihood of new treatments proving superior to standard treatments
23 it surprises us that, "as a statistician" Senn would find this evidence "unpalatable".
GoldacreB:
Bad Pharma.London: 4
thEstate2012.
Reference SourceSennS:
Misunderstanding publication bias: editors are not blameless after all [v1; ref status: indexed,
http://f1000r.es/YvAwwD].F1000 Research.2012;1(59).
10.12688/f1000research.1-59.v1DickersinKChalmersI:
Recognising, investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the World Health Organisation.JLL Bulletin: Commentaries on the history of treatment evaluation. (
www.jameslindlibrary.org).
Reference SourceTimmerAHilsdenRJColeJ:
Publication bias in gastroenterological research - a retrospective cohort study based on abstracts submitted to a scientific meeting.BMC Med Res Methodol.2002;2:7.
1197818310.1186/1471-2288-2-7115863GodleeFDickersinK:
Bias, subjectivity, chance, and conflict of interest in editorial decisions.In: Godlee F, Jefferson T, eds.
Peer review in health sciences, 2nd edition. London: BMJ Books 2003.
Reference SourceOlsonCMRennieDCookD:
Publication bias in editorial decision making.JAMA.2002;287(21):2825–2828.
1203892410.1001/jama.287.21.2825EmersonGBWarmeWJWolfFM:
Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial.Arch Intern Med.2010;170(21):1934–1939.
2109835510.1001/archinternmed.2010.406DickersinKSsemandaEMansellC:
What do JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion.BMC Med Res Methodol.2007;7:44.
1789485410.1186/1471-2288-7-442121101SchererRWLangenbergPVon ElmE:
Full publication of results initially presented in abstracts.Cochrane Database Syst Rev.2007;18(2):MR000005.
1744362810.1002/14651858.MR000005.pub3SterlingTD:
Publication decisions and their possible effects on inferences drawn from tests of significance - or vice versa.J Am Statistical Assoc.1959;54:30–34.
10.1080/01621459.1959.10501497SmartRG:
The importance of negative results in psychological research.Can Psychologist.1964;5(4):225–232.
10.1037/h0083036ChalmersTCKoffRSGradyGF:
A note on fatality in serum hepatitis.Gastroenterology.1965;49:22–26.
Reference SourceLightRJPillemerDB:
Summing up.Cambridge: Harvard University Press1984.
Reference SourceSongFParekhSHooperL:
Dissemination and publication of research findings: an updated review of related biases.Health Technol Assess.2010;14(8): iii, ix-xi,1–193.
20181324GreenwaldAG:
Consequences of prejudice against the null hypothesis.Psychol Bull.1975;82(1):1–20.
10.1037/h0076157CoursolAWagnerEE:
Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias.Prof Psychol: Res Pract.1986;17(2):136–137.
10.1037/0735-7028.17.2.136DickersinKChanSChalmersTC:
Publication bias and clinical trials.Control Clin Trials.1987;8(4):343–53.
344299110.1016/0197-2456(87)90155-3ShadishWRDohertyMMontgomeryLM:
How many studies are in the file drawer? An estimate from the family/marital psychotherapy literature.Clin Psychol Rev.1989;9(5):589–603.
10.1016/0272-7358(89)90013-5MahoneyMJ:
Publication prejudices: An experimental study of confirmatory bias in the peer review system.Cog Ther Res.1977;1(2):161–175.
10.1007/BF01173636PetersDCeciS:
Peer review practice of psychologic journals: The fate of published articles submitted again.Behav Brain Sci.1982;5(2):187–195.
10.1017/S0140525X00011183EpsteinWM:
Confirmational response bias among social work journals.Sci Technol Hum Values.1990;15(1):9–38.
10.1177/016224399001500102SavovićJJonesHEAltmanDG:
Influence of reported study design characteristics on intervention effect estimates from randomized controlled trials.Ann Intern Med.2012;157(6):429–438.
2294583210.7326/0003-4819-157-6-201209180-00537DjulbegovicBKumarAGlasziouPP:
New treatments compared with established treatments in randomized trials.Cochrane Database Syst Rev.2012;10:MR000024.
2307696210.1002/14651858.MR000024.pub3349022610.5256/f1000research.653.r651Reviewer response for version 1JuliousSteven1Referee
Medical Statistics Group, School of Health and Related Research, University of Sheffield, Sheffield, UK
Competing interests: No competing interests were disclosed.
I would just put one anecdotal observation and that is of second studies that replicate the findings of a study published in a journal. An editor may turn down the second study as 'nothing new' is being said although most would argue replication to be important.
Reviewer Expertise:
NA
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
10.5256/f1000research.653.r601Reviewer response for version 1NaldiLuigi1Referee
Department of Dermatology, Centro Studi GISED, Bergamo, Italy
Competing interests: No competing interests were disclosed.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
10.5256/f1000research.653.r600Reviewer response for version 1de VetRiekie1Referee
EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
Competing interests: No competing interests were disclosed.
The authors comment on a article by Stephen Senn who questions Ben Goldacre’s assertion in the book “Bad Pharma” that editorial process is not the main cause of publication bias. They present a large amount of evidence from the literature that researchers are the main cause of publication bias by selectively submitting paper for publication.
They provide a lot of convincing information in this short reaction. However, some sentences are very difficult to read. Especially for readers who haven’t read the book by Goldacre, the comment by Senn, and some of the other references. I had to reread the first sentence about five times before I understood. The sentence is especially difficult to read because there is a double negation. Splitting the sentence in the statement of Ben Goldacre and the comment of Stephen Senn may help. Also the last sentence of the comment is difficult to understand, especially when the reader is unaware of the conclusion of reference 23.
The second part of the citation of Goldacre “the prospects for disentangling cause and effect when it comes to publication bias are not great” is difficult to understand and, as far as I can see, does not come back in the comment. Consider whether that part can be omitted, or refer to it again at the end of the comment.
The last section starts with ‘The controlled experiments’. It is not clear to which experiments this refers. To ‘studies – including controlled experiments ‘mentioned in the first section?
In conclusion, this is a very important and informative comment. However, the readability should be improved in order to make it better understandable for readers who have not read all previous papers.
Reviewer Expertise:
NA
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.