A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines

David Blanco; Darko Hren; Jamie J. Kirkham; Erik Cobo; Sara Schroter

doi:10.12688/f1000research.20556.1

Home Browse A survey exploring biomedical editors’ perceptions of editorial interventions...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines

[version 1; peer review: 1 approved, 2 approved with reservations]

David Blanco ^1,2, Darko Hren³, Jamie J. Kirkham⁴, Erik Cobo¹, Sara Schroter⁵

David Blanco ^1,2, Darko Hren³, [...] Jamie J. Kirkham⁴, Erik Cobo¹, Sara Schroter⁵

PUBLISHED 24 Sep 2019

Author details Author details

¹ Statistics and Operations Research Department, Universitat Politècnica de Catalunya, Barcelona, Spain
² Université de Paris, CRESS, INSERM, INRA, Paris, France
³ Faculty of Humanities and Social Sciences, University of Split, Split, Croatia
⁴ Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
⁵ The BMJ, London, UK

David Blanco
Roles: Conceptualization, Data Curation, Formal Analysis, Methodology, Writing – Original Draft Preparation

Darko Hren
Roles: Conceptualization, Formal Analysis, Methodology, Writing – Review & Editing

Jamie J. Kirkham
Roles: Conceptualization, Writing – Review & Editing

Erik Cobo
Roles: Conceptualization, Writing – Review & Editing

Sara Schroter
Roles: Conceptualization, Formal Analysis, Methodology, Writing – Original Draft Preparation

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

Abstract

Background: Improving the completeness of reporting of biomedical research is essential for improving its usability. For this reason, hundreds of reporting guidelines have been created in the last few decades but adherence to these remains suboptimal. This survey aims to inform future evaluations of interventions to improve adherence to reporting guidelines. In particular, it gathers editors’ perceptions of a range of interventions at various stages in the editorial process.
Methods: We surveyed biomedical journal editors that were knowledgeable about this topic. The questionnaire included open and closed questions that explored (i) the current practice of their journals, (ii) their perceptions of the ease of implementation and the potential effectiveness of different interventions, (iii) the barriers and facilitators associated with these interventions, and (iv) suggestions for future interventions and incentives.
Results: Of the 99 editors invited, 24 (24%) completed the survey. Involving trained editors or administrative staff was deemed the potentially most effective intervention but, at the same time, it was considered moderately difficult to implement due to logistic and resource issues. Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and could decrease the overall quality of reviews. Journals incentivising adherence, and publishers and medical institutions encouraging journals to adopt strategies to boost adherence were two recurrent themes.
Conclusions: Further evaluation of interventions are required. These evaluations could take into account the points raised in this survey.

Keywords

Completeness of reporting, Journal policies, Quality of reporting, Reporting guidelines, Survey, Barriers, Facilitators

Corresponding author: David Blanco

Competing interests: SS is Senior Researcher at The BMJ.

Grant information: This study belongs to the ESR 14 research project from the Methods in Research on Research (MiRoR) project, which has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2019 Blanco D et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Blanco D, Hren D, Kirkham JJ et al. A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines [version 1; peer review: 1 approved, 2 approved with reservations]. F1000Research 2019, 8:1682 (https://doi.org/10.12688/f1000research.20556.1) First published: 24 Sep 2019, 8:1682 (https://doi.org/10.12688/f1000research.20556.1) Latest published: 23 Dec 2019, 8:1682 (https://doi.org/10.12688/f1000research.20556.3)

Abbreviations

RGs: reporting guidelines; CONSORT: Consolidated Standards of Reporting Trials; RCT: Randomised controlled trials; EQUATOR: Enhancing the QUAlity and Transparency Of Health Research; MiRoR: Methods in Research on Research; STROBE: STrengthening the Reporting of OBservational studies in Epidemiology; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; APCs: article processing charges; CME: continuing medical education; ICJME: International Committee of Medical Journal Editors.

Introduction

Transparent and accurate reporting of research is essential for increasing the usability of available research evidence¹. Reporting guidelines (RGs) can be useful tools to help authors report research methods and findings in a way that they can be understood by readers, replicated by researchers, used by health care professionals to make clinical decisions, and included in systematic reviews¹. Since the inception in 1996 of the Consolidated Standards of Reporting Trials (CONSORT) for the reporting of randomised controlled trials (RCTs)², more than 400 RGs for different study types, data, and clinical areas have been developed. These RGs can be found in the library of the Enhancing the Quality and Transparency Of Health Research (EQUATOR) Network¹.

Biomedical authors’ adherence to RGs has been observed to be suboptimal³. Consequently, in recent years various stakeholders have proposed, and sometimes evaluated, the impact of different types of interventions to improve this adherence. These interventions were identified and classified in a recently published scoping review⁴. We found that the strategies most widely used by journals have been shown not to have the desired effect^5–8 and this highlighted the need for the implementation and evaluation of the other interventions proposed⁴.

This paper reports a survey aimed to inform the future evaluation of interventions to improve adherence to RGs. In particular, we focused on interventions that can be implemented at various points in the editorial process. Our specific objectives were to explore the perceived ease of implementation and potential effectiveness of various interventions; to map the barriers and facilitators associated with these interventions; to determine possible solutions to overcome the barriers described, and to identify further editorial interventions that could be implemented and subsequently evaluated.

Methods

Participants

Purposive sampling was used to recruit biomedical editors that were knowledgeable about the topic we aimed to explore. Participants were sampled from three sources: (i) editors of journals that had published studies describing interventions to improve adherence to RGs identified in our scoping review⁴, (ii) members of the Methods in Research on Research (MiRoR) Network with current editorial positions and (iii) editors of the top-10 journals (based on impact factor) of BMJ Publishing Group which, apart from being one of the partner institutions of MiRoR, has published the main RGs^9–12 and has traditionally performed research to improve the transparency and quality of biomedical publications¹³. The authors of this survey who met the eligibility criteria were excluded as potential participants.

Procedure

To contact editors not known to us we sought email addresses in the public domain. Three editors (including the editors-in-chief) of each of the sampled journals, as well as individual editors from the group (ii) above, were sent a personalised email inviting them to complete an online survey investigating their opinions about different editorial interventions to improve author adherence to RGs. The survey was administered by SurveyMonkey and was open between 27 November 2018 and 24 February 2019. Each survey invitation was tied to a unique email address. Two reminders to complete the survey were sent to non-responders at four and eight weeks after the initial mailing. Participants could edit their responses while completing the survey, but not re-enter the survey once it was completed. We recorded how many people opened the invitation or clicked through to the survey, as well as the number of surveys completed. Participants could suggest further editors that they considered could contribute to the survey. These participants were also sent a personalised invitation.

Questionnaire development

Our previous scoping review⁴ identified 31 interventions targeting different stakeholders in the research process. For use in this survey we chose a smaller subset of nine interventions that could be implemented during the editorial process as our focus was on journal editors’ perceptions (see Box 1).

Box 1. Interventions included and their targets.

A. Interventions targeting authors:
- • A requirement for authors to submit a completed RG checklist (using all appropriate extensions, if applicable) indicating the page numbers where each item is addressed (Intervention 1)
- • A requirement for authors to submit a populated RG checklist with text from their manuscript in order to facilitate the peer review process (Intervention 2)
- • A requirement for authors to highlight in the manuscript where each RG item is addressed (Intervention 3)
- • A requirement for authors to include new subheadings within their manuscript corresponding to different RG items within the traditional IMRaD format (Introduction, Methods, Results, and Discussion) (Intervention 4)
- • A requirement for authors on submission to use a freely available writing aid tool that guides authors through the RG checklist items, shows the key elements that need to be reported, and includes examples of adequate reporting (e.g. COBWEB) (Intervention 5)
B. Interventions targeting peer reviewers:
- • Instruct peer reviewers to use the appropriate RGs when assessing a manuscript (Intervention 6)
- • Instruct peer reviewers to scrutinise the completed RG checklist submitted by the authors and check its consistency with the information reported in the manuscript (Intervention 7)
C. Interventions targeting editorial staff:
- • An evaluation of the completeness of reporting by a trained editor (or editorial assistant), who would return incomplete manuscripts to authors before considering the manuscript for publication (Intervention 8)
D. Interventions targeting authors, peer reviewers, and editors:
- • Training for authors, peer reviewers, and editors on the importance, content, and use of RGs (e.g. The EQUATOR Network toolkits) (Intervention 9)

The survey combined open and closed response questions to seek participants’ perceptions of a series of interventions to improve authors’ adherence to RGs that could potentially be implemented during the editorial process. We pilot tested the draft survey questionnaire with two collaborators of the MiRoR project who currently hold editorial positions. They were asked to review the survey for its clarity and completeness and to provide suggestions on how to improve its structure.

Based on feedback from the pilot we decided not to include the intervention “Implementation of the automatic tool Statreviewer¹⁴” since participants were not aware of this software and stated that their perceptions would strongly depend on details about how it operates which are not publicly available.

We structured the final questionnaire (see Figure S1, Extended data)¹⁵ as follows:

Part 1: Current practice. Participants were asked to describe the measures their journal currently takes to improve adherence to RGs.
Part 2: Perceptions of nine potential interventions. Participants were asked to indicate on 5-point Likert scales (i) how easy it would be (or was) to implement these interventions at their journals (1-very difficult, 2-moderately difficult, 3-neither difficult nor easy, 4-moderately easy, 5-very easy) and (ii) how effective they thought the interventions would be (or was) at improving adherence to RGs if these were implemented at their journals (1-very ineffective, 2-moderately ineffective, 3-neither ineffective nor effective, 4-moderately effective, 5-very effective). We included images to clarify meanings and context to prompt participants to think about the benefits and drawbacks of the interventions. Free text boxes were included so participants could justify their responses.
Part 3: Identifying the barriers and facilitators. Participants were asked to choose which intervention they considered potentially the most effective for their journal. They were asked to describe (i) why they thought that intervention would be the most effective, (ii) what the main difficulties in implementing that intervention would be, and (iii) how they would try to overcome these difficulties.
Part 4: Further interventions. Participants were asked for further suggestions of possible interventions, including modifications and combinations of the interventions previously discussed.
Part 5: Demographic questions.

Data analysis

Descriptive statistics were calculated for quantitative data using R version 3.6.0¹⁶. For qualitative information, the lead investigator (DB) used the software program NVivo 12¹⁷ to extract data and classify the data into key themes. This classification was discussed with another investigator (SS) and subsequently refined.

For Part 1 of the survey (Current practice) the unit of measure were the journals and therefore editors of the same journal were grouped. For all other parts of the survey, we analysed editors’ responses independently, no matter what their journal was.

Ethics approval & informed consent

The Research Committee of the Governing Council of the Universitat Politècnica de Catalunya (UPC) granted ethical approval for this study (Reference EC 01, Date 2 May 2018).

Participants were informed that completion of the survey indicated consent to participate, and that they were free to stop and withdraw from the study at any time without providing a reason.

Reporting guidelines

We consulted the Checklist for Reporting of Results of Internet E-Surveys (CHERRIES)¹⁸ and the Consolidated criteria for Reporting of Qualitative research (COREQ)¹⁹ guidelines to produce this research report.

Results

Of the 99 editors invited, 42 opened the invitation (view rate 42%), and 24 completed the survey (response rate 24%) from the 25 who started it (completion rate 96%). The average time spent completing the survey was 15 minutes (SD = 8.5 minutes). The 24 participants were editors of 20 different biomedical journals and had a variety of editorial roles (editor-in-chief, senior editor, associate editor or others). Most of them were involved in manuscript decision-making and had less than 15 years of experience as journal editors. Table 1 shows their demographic characteristics. Raw survey results are given as Underlying data²⁰.

Table 1. Demographic characteristics of the 24 participants.

N=24
Current position	Working full time as a journal editor	8 (33%)
	Working part time (equal or more than 0.5 of their time) as a journal editor	1 (4%)
	Working part time (less than 0.5 of their time) as a journal editor	14 (59%)
	Other (Volunteer editor)	1 (4%)
Editorial role	Editor-in-chief	10 (41%)
	Senior editor	4 (17%)
	Associate editor	4 (17%)
	Other (Editorial director, Technical editor, Assistant editor)	6 (25%)
Involvement in manuscript decision- making	Yes	22 (92%)
Involvement in manuscript decision- making	No	2 (8%)
Years of experience as a journal editor	<5	8 (33%)
	5–15	12 (50%)
	15–25	3 (13%)
	>25	1 (4%)

Current practice

Respondents worked at 20 journals. Most respondents’ journals (12/20, 60%) request authors to submit a completed RG checklist with page numbers indicating where the items are addressed when they submit their manuscript. A further seven (35%) instruct but do not request authors to do it, and one (5%) does not request or instruct authors. Among the journals requesting the submission of checklists, four (4/12, 33%) also explicitly ask peer reviewers to use the completed RGs when assessing manuscripts, one (1/12, 8%) asks peer reviewers general questions about the completeness of reporting, and one performs an evaluation of the completeness of reporting by a trained editor using RGs before the initial decision is made on the paper. We observed no incongruences between the answers of editors from the same journal. Some respondents mentioned that in their journals (n=4) the interventions described were only applicable to the study types corresponding to the most established RGs (CONSORT, STROBE, or PRISMA) for trials, observational studies and systematic reviews respectively.

Perceptions of nine potential interventions

The mean scores for perceived ease of implementation and potential effectiveness for each intervention are shown in Figure 1.

Figure 1. Scores for perceived ease of implementation and perceived effectiveness.

White dots represent the mean scores for each of the categories. Interventions in red target authors, those in grey target peer reviewers, the one in orange target editors or administrative staff and the one in blue targets all these stakeholders. An explanation of the content of each intervention can be found in Box 1.

The two most common interventions were considered the easiest ones to implement: the mean scores for requesting authors to submit checklists with page numbers (Intervention 1) and for asking peer reviewers to use RGs (Intervention 6) were 4.33 (SD=0.90) and 3.67 (SD=1.14), respectively. By contrast, interventions related to training (Intervention 9), editor involvement in checking completeness of reporting (Intervention 8) and reformatting of the text based on RG requirements (Intervention 4, Intervention 5) were considered the most difficult to implement.

An evaluation of the completeness of reporting by a trained editor was considered the most effective intervention (4.09, SD=1.02) and the two targeting peer reviewers (Interventions 6 and 7) were perceived as being the least effective (3.13, SD=1.17; 2.96, SD=1.06). All interventions targeting authors (Interventions 1-5) and training (Intervention 9) ranged between 3.3 and 3.6.

Identifying the barriers and facilitators

This section presents the perceived barriers and facilitators of the interventions considered and editors’ suggestions for making the interventions more effective. Table S1 in Extended data¹⁵ shows a full description of these.

A) Interventions targeting authors (1-5)

The main barriers associated with all of the interventions targeting authors was that authors have to state their adherence to the relevant RG and this does not equate to actual compliance. Moreover, it is resource intensive for journals to check that these requirements are appropriately met by authors. Some editors highlighted that Interventions 3, 4, and 5 would involve special formatting of the submitted manuscript, which could be cumbersome for authors given that manuscripts are often submitted to multiple journals with different formats before being accepted. This is particularly relevant for journals with high rejection rates as it could cause frustration for authors. Some participants mentioned logistical issues as their journal’s manuscript tracking system is not set up to accommodate these interventions. In addition, changes in the manuscript’s format could be incompatible with the journal’s house style.

Intervention 1 was generally considered quick and straightforward for authors, but several participants indicated that there is published empirical evidence of little effectiveness if the checklist is not assessed by a trained editor or administrator^5–8.

As Interventions 3, 4, and 5 force authors to tailor the manuscript to RG requirements, participants reported that these could make editors’ and peer reviewers’ jobs easier as the manuscript would be better structured. Importantly, readers would also be able to locate information more easily. Some editors pointed out that, to make these interventions effective, journals would need to provide templates to authors or to integrate these interventions in the submission system. However, some of these interventions (Interventions 2 and 5) were seen as more effective if they were implemented earlier on in the research process, prior to writing the manuscript.

B) Interventions targeting peer reviewers (6, 7)

Most respondents were negative about the potential effectiveness of implementing the two interventions targeting peer reviewers (Intervention 6 and 7) as they felt these would create too much additional work for reviewers. Participants were concerned that the quality of peer review could be compromised as reviewers should focus on the manuscript’s content and not on the reporting issues. Furthermore, peer reviewers may not know which RGs to use and, even if they do, the effectiveness would be dependent on their willingness to use RGs and their expertise in applying them. Several participants indicated that this work should be delegated to paid editorial staff.

C) Interventions targeting editorial staff (8)

This intervention was considered difficult to implement but potentially effective. The main facilitating factor for its successful implementation was that it is performed by a paid or trained professional, which lends credibility to the intervention, reduces the workload of unpaid peer reviewers, and avoids authors overclaiming adherence. The main barriers outlined for this intervention were (i) the budget issues the journal would need to face to train or hire additional editorial staff that could perform the evaluation, especially if the journal receives a large volume of papers, (ii) the editorial delays it may cause, and the (iii) the potential inefficiency of assistant editors or administrators having to delegate decisions in case of doubt, given that sometimes assessing completeness of reporting is a subjective task.

To make this intervention more feasible for journals, editors suggested that the completeness of reporting evaluation could be performed only for papers that are sent out for peer review and, it could be focused on a few core items (different for each RG) that would enable reproducibility. If this intervention was implemented in a journal that requires the submission of a completed checklist, editors could take advantage of the checklist to locate information.

D) Interventions targeting authors, peer reviewers and editors (9)

Training was seen as a potentially effective intervention but difficult to implement. Some participants highlighted that training with follow up sessions would be resource intensive for journals, and especially difficult to enforce. One participant mentioned that credits (such as CME credits²¹) could be used to recognise hours of training. The fact that sometimes the editorial staff is based in different places and zones makes it crucial to consider flexible forms of training, such as online courses. As an example, the EQUATOR Network Toolkits section provides resources for authors, peer reviewers and journal editors²². However, some participants emphasised that training should also be delivered by research institutions and medical centres.

Further interventions and incentives for authors and journals

The implementation of reading tools that automatically assess adherence to RGs, such as Statreviewer¹⁴, were seen as potentially interesting interventions. Some respondents also mentioned the possibility of combining some of the interventions listed, such as requiring the submission of checklists and trained editors assessing the responses with the information reported in the manuscript.

Moreover, several incentives for authors were listed, including (i) discounts on article processing charges (APCs) for authors that comply with RG requirements, (ii) academic institutions including RG use in the promotion and tenure files, and (iii) credits (such as CME credits²¹) to recognise hours of training on the use of RGs. Journals could also be encouraged to implement certain interventions if (i) there is empirical evidence that these interventions actually improve the reporting quality of the papers or (ii) publishers or the International Committee of Medical Journal Editors (ICMJE) mandate these as a condition of submission to their journals. Even if some of these interventions are proven to be effective, some respondents reported that it is essential to convince publishers that improving the quality of reporting is a worthy investment to resource.

Discussion

This survey explores biomedical journal editors’ perceptions of the practical aspects of the implementation of different interventions to improve adherence to RGs.

Several messages arise from this study. First of all, most editors agreed that the most effective way to improve adherence to RGs is for journals to involve trained editors or administrative staff. Interventions targeting these stakeholders were considered to be difficult to implement for most journals, either because of logistic or resource issues. However, improving the performance of editorial staff is critical²³ and has been shown to have a positive impact on completeness of reporting in the context of a dentistry journal²⁴. To make these type of interventions more feasible, journals could implement them only for manuscripts that are sent out for peer review. The editorial staff could also take advantage of the RG checklists submitted by authors, that could be automatically populated with text using specific software such as the the tool proposed by Hawwash et al.²⁵

Most editors considered that checking reporting issues is beyond the role of peer reviewers. Given the voluntary nature of peer review, requiring reviewers to use RGs causes an additional workload that could compromise the overall quality of the reviews. Furthermore, as finding peer reviewers is becoming increasingly difficult for editors²⁶, these requirements could make them even less willing to review papers. Additionally, some editors considered that the average peer reviewer does not have enough expertise to go over RG requirements.

We observed that the interventions perceived as potentially most effective appear to be more difficult to implement. Conversely, the most common strategies seem to have been implemented based on their feasibility and not on their potential to improve completeness of reporting. This could be one of the reasons why they have failed to achieve the desired results^5–8. Some of our respondents insisted that a key element is that journals, universities, and medical institutions find ways to incentivise author’s compliance with RGs. At the same time, the scientific community needs to find ways to convince publishers that improving the quality of reporting is a worthy investment so that publishers can encourage their journals to adopt strategies to boost completeness of reporting. A recent article indicates that implementing RGs through the editorial process may increase the number of citations to the research reported²⁷.

A common observation by the survey participants was that the effectiveness of the interventions proposed could depend on the types of articles considered. While RGs for randomised trial protocols, randomised trials or systematic reviews are more established, some others, including most RG extensions, are not well known to the stakeholders involved in the publication process. For this reason, it is important for journals to be clear in their “Instructions for Authors” on what RGs they mandate.

We encourage researchers to perform further evaluations of interventions in collaboration with biomedical journals, such as the RCT our research team is currently undergoing²⁸. Our study aims to evaluate the effect on completeness of reporting of a trained researcher assessing during peer review the consistency between the CONSORT checklists submitted by authors and the information reported in the manuscript, and providing authors with a report indicating any inconsistencies found.

Providing high quality evidence of the effectiveness of different interventions to improve adherence to RGs and discussing how to make them less burdensome are key aspects needed to convince all stakeholders that this effort is worth it.

Data availability

Underlying data

Zenodo: Underlying data of the project “A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines”. DOI: https://doi.org/10.5281/zenodo.3407725²⁰.

This project contains the following underlying data:

Survey dataset (Dataset including all survey responses).

Extended data

Zenodo: Extended data of the project “A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines”. https://doi.org/10.5281/zenodo.3404002¹⁵.

This project contains the following extended data:

Figure S1: Survey questionnaire (Complete version of the survey questionnaire used in this project)
Table S1: Barriers, facilitators and possible improvements of the included interventions (Table containing the barriers, facilitators and possible improvements identified for each of the interventions explored in the survey)

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Acknowledgments

The authors thank the MiRoR Project and Marie Sklodowska-Curie Actions for their support. This survey is the second part of a larger project whose first part was a scoping review to identify and classify interventions to improve adherence to RGs⁴. The third part is an RCT to evaluate the impact of assessing during peer review the CONSORT checklist submitted by authors²⁸.

Faculty Opinions recommended

References

1. EQUATOR Network: EQUATOR Network website-the resource centre for. 2013. Reference Source
2. Begg C, Cho M, Eastwood S, et al.: Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996; 276(8): 637–9. PubMed Abstract | Publisher Full Text
3. Samaan Z, Mbuagbaw L, Kosa D, et al.: A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013; 6: 169–188. PubMed Abstract | Publisher Full Text | Free Full Text
4. Blanco D, Altman D, Moher D, et al.: Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open. 2019; 9(5): e026589. PubMed Abstract | Publisher Full Text | Free Full Text
5. Shamseer L, Stevens A, Skidmore B, et al.: Does journal endorsement of reporting guidelines influence the completeness of reporting of health research? A systematic review protocol. Syst Rev. 2012; 1(1): 24. PubMed Abstract | Publisher Full Text | Free Full Text
6. Stevens A, Shamseer L, Weinstein E, et al.: Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review. BMJ. 2014; 348: g3804. PubMed Abstract | Publisher Full Text | Free Full Text
7. Blanco D, Biggane AM, Cobo E, et al.: Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers? Trials. 2018; 19(1): 80. PubMed Abstract | Publisher Full Text | Free Full Text
8. Hair K, Macleod MR, Sena ES, et al.: A randomised controlled trial of an Intervention to Improve Compliance with the ARRIVE guidelines (IICARus). Res Integr Peer Rev. 2019; 4(1): 12. PubMed Abstract | Publisher Full Text | Free Full Text
9. Schulz KF, Altman DG, Moher D, et al.: CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010; 340: c332. PubMed Abstract | Publisher Full Text | Free Full Text
10. Moher D, Liberati A, Tetzlaff J, et al.: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009; 339: b2535. PubMed Abstract | Publisher Full Text | Free Full Text
11. von Elm E, Altman DG, Egger M, et al.: Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007; 335(7624): 806–8. PubMed Abstract | Publisher Full Text | Free Full Text
12. Chan AW, Tetzlaff JM, Gøtzsche PC, et al.: SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013; 346: e7586. PubMed Abstract | Publisher Full Text | Free Full Text
13. Evidence based publishing. Reference Source
14. Statreviewer. Reference Source
15. Blanco D: Extended data of the project "A survey exploring biomedical editors' perceptions of editorial interventions to improve adherence to reporting guidelines" [Data set]. Zenodo. 2019. http://www.doi.org/10.5281/zenodo.3404002
16. R: A language and environment for statistical computing. 2019. Reference Source
17. NVivo qualitative data analysis software. 2018. Reference Source
18. Eysenbach G: Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004; 6(3): e34. PubMed Abstract | Publisher Full Text | Free Full Text
19. Tong A, Sainsbury P, Craig J: Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Heal Care. 2007; 19(6): 349–57. PubMed Abstract | Publisher Full Text
20. Blanco D: Underlying data of the project "A survey exploring biomedical editors' perceptions of editorial interventions to improve adherence to reporting guidelines" [Data set]. Zenodo. 2019. http://www.doi.org/10.5281/zenodo.3407725
21. Health NI of: What is CME Credit? Reference Source
22. EQUATOR Network Toolkits. Reference Source
23. Cobo E, González JA: Taking advantage of unexpected WebCONSORT results. BMC Med. 2016; 14(1): 204. PubMed Abstract | Publisher Full Text | Free Full Text
24. Pandis N, Shamseer L, Kokich VG, et al.: Active implementation strategy of CONSORT adherence by a dental specialty journal improved randomized clinical trial reporting. J Clin Epidemiol. 2014; 67(9): 1044–8. PubMed Abstract | Publisher Full Text
25. Hawwash D, Lachat C, De Cock N, et al.: Integrating a writing aid to facilitate the use of reporting guidelines: A crossover randomized controlled trial Trial protocol. Reference Source
26. Publons: Global State of Peer Review 2018. Reference Source
27. Vilaró M, Cortés J, Selva-O’Callaghan A, et al.: Adherence to reporting guidelines increases the number of citations: the argument for including a methodologist in the editorial process and peer-review. BMC Med Res Methodol. 2019; 19(1): 112. PubMed Abstract | Publisher Full Text | Free Full Text
28. Blanco D, Schroter S, Moher D, et al.: Evaluating the Impact of Assessing During Peer Review the CONSORT Checklist Submitted by Authors. Reference Source

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 24 Sep 2019

Author details Author details

David Blanco
Roles: Conceptualization, Data Curation, Formal Analysis, Methodology, Writing – Original Draft Preparation

Darko Hren
Roles: Conceptualization, Formal Analysis, Methodology, Writing – Review & Editing

Jamie J. Kirkham
Roles: Conceptualization, Writing – Review & Editing

Erik Cobo
Roles: Conceptualization, Writing – Review & Editing

Sara Schroter
Roles: Conceptualization, Formal Analysis, Methodology, Writing – Original Draft Preparation

Competing interests

SS is Senior Researcher at The BMJ.

Grant information

This study belongs to the ESR 14 research project from the Methods in Research on Research (MiRoR) project, which has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (3)

version 3

Revised

Published: 23 Dec 2019, 8:1682

https://doi.org/10.12688/f1000research.20556.3

version 2

Revised

Published: 12 Dec 2019, 8:1682

https://doi.org/10.12688/f1000research.20556.2

version 1

Published: 24 Sep 2019, 8:1682

https://doi.org/10.12688/f1000research.20556.1

© 2019 Blanco D et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Blanco D, Hren D, Kirkham JJ et al. A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines [version 1; peer review: 1 approved, 2 approved with reservations]. F1000Research 2019, 8:1682 (https://doi.org/10.12688/f1000research.20556.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 1

VERSION 1

PUBLISHED 24 Sep 2019

Views

Reviewer Report 26 Nov 2019

Patricia Logullo, EQUATOR Network, University of Oxford, Oxford, UK

Approved with Reservations

https://doi.org/10.5256/f1000research.22597.r56388

Positive points:

This is a much-needed survey, and I am happy and thankful that the MiRoR team invested their time and expertise in undertaking it. The subject of the adoption of reporting guidelines in the editorial process is sensitive, relevant and worth exploring. Hearing editors is a crucial step for the development of interventions aimed to improve the quality of biomedical research reporting. I congratulate the authors for the initiative.

The main result of the study (that the interventions perceived as effective are the most difficult to implement and vice-versa) points out to the need to verifying feasibility before proposing interventions, something that we should all think about.

I hope the authors find these suggestions helpful and constructive, and that they find these to be positive contributions.

Major concerns: methodology

1) Sample characteristics. Although the authors should be praised for the idea of asking editors about interventions to improve reporting, I have a concern about what editors should be surveyed. If one wants to explore the perceived ease of the implementation of one intervention, is it adequate to choose a population of editors whom we know are already “knowledgeable about the topic”?

If these editors are aware of the reporting quality crisis and the reporting guidelines in general, wouldn’t these be more prone to adopt interventions that aim to improve the quality of publications? Therefore, aren’t these the people who would say “yes” to any intervention proposed?

Should the authors not interview editors who have never heard about reporting guidelines too? Wouldn’t this heterogeneous sample be more representative of the scenario of difficulties — or the ease — of implementing reporting guidelines checks in the editorial process? Are the study results only applicable to journals that already endorse or require reporting guidelines?

I would like to see comments about this limitation in the discussion if the authors agree with this or even if they do not.

2) Sample size. My second most important concern is about sample size. In the Methods section, the authors do not explain how they planned the sample size and based on what assumptions. 24% seems to be a very low response rate. Do the authors believe 24 is enough? This limitation should be discussed.

3) Effectiveness. When you asked your participants about their opinions on “how effective” something would be, what was the definition of effectiveness? Would it be their perception about the intervention improving the adherence to reporting guidelines checklists? Was it a more subjective view of “manuscript quality”?

4) Statreviewer. The authors mention in the text, in Methods:
“Based on feedback from the pilot we decided not to include the intervention “Implementation of the automatic tool Statreviewer14” since participants were not aware of this software and stated that their perceptions would strongly depend on details about how it operates which are not publicly available.”
However, on page 7, Results, they state:
“The implementation of reading tools that automatically assess adherence to RGs, such as Statreviewer14, were seen as potentially interesting interventions.”

After all, was this intervention Statreviewer evaluated (as in Results) or was it not (as in Methods)? Do you mean they mentioned other tools, such as Penelope, GoodReports, Cobweb or others? Or were they referring to Statreviewer?

Major concerns: reporting

5) Journals. You interviewed 24 editors from 20 journals. What are these journals? I did not see a list of their names, and I believe it is important.

6) Figure 1. Figure 1 is difficult to read/interpret. I like the idea of having the two outcomes (ease of implementation and effectiveness) in one figure very much so that one can compare the two. However, the bar chart is not clear at all, nor easy to interpret. Colours do not help. The authors should consider building a figure with plot points (which would be easy, given the small sample) showing where each participant sits within the Likert scale — not bars.

Another suggestion for Figure 1 (here, just a suggestion) is to add something to remind the readers what the interventions are. As Box 1 is far from the Figure, it is difficult to remember what each intervention is. It would be useful to have “nicknames”, for example:
1. Page numbers
2. Checklist
3. Highlight
4. Subheadings
5. COBWEB
6. Use RG
7. Check RG
8. Staff
9. Training

7) Peer reviewers’ checks. The abstract is, in general, well written and clear. However, I got intrigued by this sentence:
“Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and could decrease the overall quality of reviews.”

I was surprised that peer reviewers checking reporting guidelines adherence would decrease the overall quality of the paper. Then I reread it in the paper, but here put in a complete sentence, that makes sense:

“Participants were concerned that the quality of peer review could be compromised as reviewers should focus on the manuscript’s content and not on the reporting issues.”

Meaning, I suppose, that peer reviewers would focus on checking reporting guidelines and not on their “expert view” of the study. I get it, but it is not very clear, neither in the text, nor in the abstract, and I invite the authors to rewrite both. Especially because “reporting guidelines” are about content too. They point out the content elements that should not be absent from papers, so using the word “content” in the manuscript text is not really appropriate. Please, re-evaluate.

8) As I said above, the abstract is good, and there is room for growth (you have now 232 words). However, I could not find the conclusions in the abstract helpful. Could you please inform better what you did conclude? What “points raised in the survey” should be considered?

9) Checklists as forms. ...“assessing completeness of reporting is a subjective task”. This is one of the most important sentences in this manuscript, it tells us about using reporting guidelines checklists as evaluation tools — for what they have not been created. I saw no discussion about this in your paper.

10) Discussion. The discussion of the paper is short, and it lacks several points pointed above. Mainly, it does not discuss the limitations and potential bias of this study. Discussing limitations is a requirement from F1000Research journal.

11) You mention you did analyse qualitative information using NVivo. How did you do it?

12) There are at least eight items from the CHERRIES checklist that I could not see in your paper, and you might want to report, even if in a separate (or supplementary) box, from data protection, preventing multiple entries, to randomisation of items, and others.

13) From Box 1, I do not quite get the difference between Interventions 1 and 2. It seems that both ask authors to submit a populated/completed RG checklist. Is it that #1 asks for page numbers and #2 does not?

14) In my opinion, the sentences from “To contact editors...” to “improve adherence to RGs” belong more to the “Participants” section than to the “Procedure”. Especially the first sentence.
Also “Participants could suggest further editors that they considered could contribute to the survey” is something that I would consider as a snowballing or recruitment method, not the survey procedure per se. So, I would suggest putting everything related to recruitment together in one section, to make it easier for your reader (who might be interested in undertaking a similar survey) to find information on such a critical issue as recruitment. Also: how could participants suggest? Was it a specific question in the SurveyMonkey form?

15) Were participants given an estimation of the time required to answer the survey before they began?

Minor: about citations

16) Reference number 1 (EQUATOR Network website) is not a proper, complete reference. It points to the home page of the website, not to a document, and as so, it does not show from where the authors collected that information (the reader cannot find them in the home page). The correct way of citing would be adding the full URL. A suggestion: EQUATOR Network. What is a reporting guideline? Available at:
http://www.equator-network.org/about-us/what-is-a-reporting-guideline/. Cited on 19/11/2019.

17) Something similar happens to reference number 13. Author and title are missing. It would help readers if it could be typed like this:
Schoter S. Evidence-based publishing. BMJ. Available at: https://www.bmj.com/about-bmj/evidence-based-publishing. Cited on 19/11/2019.

18) Reference 14 also lacks authors and titles. References 13, 14, 16, 21 and 22 lack a lot of elements necessary for readers to be able to find them. When working in a printed version (as I am now), you cannot see what the document is, where it is published or by whom — these are available only by those reading online and able to “click” on the links.

19) I would change the sentence:
“Raw survey results are given as Underlying data²⁰.”
To something clearer and that does not require the reader to go to the reference list to understand what it is.
“Table XX in the supplementary file X²⁰ shows the responses from all 24 participants.”

20) Has the study by Hawwash et al. (reference 25) been published yet? (The authors cited the protocol only). Do the authors plan to cite other electronic tools that are out there?

Thank you for the opportunity of reading and evaluating this paper.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

I cannot comment. A qualified statistician is required.
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Methodology. Reporting quality. Research integrity.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 12 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

12 Dec 2019

Author Response

We thank you for your time and detailed comments on the manuscript. Please find below a point by point response to them:

Major concerns: methodology

1) Sample characteristics. Although the authors should ... Continue reading We thank you for your time and detailed comments on the manuscript. Please find below a point by point response to them:

Major concerns: methodology

1) Sample characteristics. Although the authors should be praised for the idea of asking editors about interventions to improve reporting, I have a concern about what editors should be surveyed. If one wants to explore the perceived ease of the implementation of one intervention, is it adequate to choose a population of editors whom we know are already “knowledgeable about the topic”?
If these editors are aware of the reporting quality crisis and the reporting guidelines in general, wouldn’t these be more prone to adopt interventions that aim to improve the quality of publications? Therefore, aren’t these the people who would say “yes” to any intervention proposed?
Should the authors not interview editors who have never heard about reporting guidelines too? Wouldn’t this heterogeneous sample be more representative of the scenario of difficulties — or the ease — of implementing reporting guidelines checks in the editorial process? Are the study results only applicable to journals that already endorse or require reporting guidelines?
I would like to see comments about this limitation in the discussion if the authors agree with this or even if they do not.

Firstly, we have now included some more information in the “Participants” section to make it clear what kind of participants we were aiming to reach. Secondly, we have discussed in the Discussion section the other points you mention.

2) Sample size. My second most important concern is about sample size. In the Methods section, the authors do not explain how they planned the sample size and based on what assumptions. 24% seems to be a very low response rate. Do the authors believe 24 is enough? This limitation should be discussed.
We have now discussed these points in the Discussion section.

3) Effectiveness. When you asked your participants about their opinions on “how effective” something would be, what was the definition of effectiveness? Would it be their perception about the intervention improving the adherence to reporting guidelines checklists? Was it a more subjective view of “manuscript quality”?
Thanks to your comment, we have just realised that we only mentioned that we were talking about “effectiveness at improving adherence to RGs” in the methods section, but not in the abstract and introduction. We have now clarified that in the abstract and introduction, and also in some parts of the methods, results and discussion to remind the readers that we refer to effectiveness at improving adherence to RGs.

4) Statreviewer. The authors mention in the text, in Methods:
“Based on feedback from the pilot we decided not to include the intervention “Implementation of the automatic tool Statreviewer14” since participants were not aware of this software and stated that their perceptions would strongly depend on details about how it operates which are not publicly available.”
However, on page 7, Results, they state:
“The implementation of reading tools that automatically assess adherence to RGs, such as Statreviewer14, were seen as potentially interesting interventions.”
After all, was this intervention Statreviewer evaluated (as in Results) or was it not (as in Methods)? Do you mean they mentioned other tools, such as Penelope, GoodReports, Cobweb or others? Or were they referring to Statreviewer?
We have clarified in the results section (“Further interventions and incentives for authors and journals”) that Statreviewer was not included in the survey but mentioned by some participants when asked about further potentially effective interventions.

Major concerns: reporting
5) Journals. You interviewed 24 editors from 20 journals. What are these journals? I did not see a list of their names, and I believe it is important.
The 20 journals represented are now listed in Table 1. We have also improved the description of the sample in the first paragraph of the Results section.

6) Figure 1. Figure 1 is difficult to read/interpret. I like the idea of having the two outcomes (ease of implementation and effectiveness) in one figure very much so that one can compare the two. However, the bar chart is not clear at all, nor easy to interpret. Colours do not help. The authors should consider building a figure with plot points (which would be easy, given the small sample) showing where each participant sits within the Likert scale — not bars.
We have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.

Another suggestion for Figure 1 (here, just a suggestion) is to add something to remind the readers what the interventions are. As Box 1 is far from the Figure, it is difficult to remember what each intervention is. It would be useful to have “nicknames”, for example:
1. Page numbers
2. Checklist
3. Highlight
4. Subheadings
5. COBWEB
6. Use RG
7. Check RG
8. Staff
9. Training
We have incorporated similar nicknames to help readers remember the interventions.

7) Peer reviewers’ checks. The abstract is, in general, well written and clear. However, I got intrigued by this sentence:
“Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and could decrease the overall quality of reviews.”
I was surprised that peer reviewers checking reporting guidelines adherence would decrease the overall quality of the paper. Then I reread it in the paper, but here put in a complete sentence, that makes sense:
“Participants were concerned that the quality of peer review could be compromised as reviewers should focus on the manuscript’s content and not on the reporting issues.”
Meaning, I suppose, that peer reviewers would focus on checking reporting guidelines and not on their “expert view” of the study. I get it, but it is not very clear, neither in the text, nor in the abstract, and I invite the authors to rewrite both. Especially because “reporting guidelines” are about content too. They point out the content elements that should not be absent from papers, so using the word “content” in the manuscript text is not really appropriate. Please, re-evaluate.
We completely agree with the point you raise. We have re-written this point in the Abstract, Results and Conclusions. “Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and were concerned that the quality of peer review could be compromised. Reviewers are generally not expected to focus on reporting issues but on providing an expert view on the importance, novelty, and relevance of the manuscript.”

8) As I said above, the abstract is good, and there is room for growth (you have now 232 words). However, I could not find the conclusions in the abstract helpful. Could you please inform better what you did conclude? What “points raised in the survey” should be considered?
We have re-written and expanded the Conclusions in the abstract. The abstract is now 295 words.

9) Checklists as forms. ...“assessing completeness of reporting is a subjective task”. This is one of the most important sentences in this manuscript, it tells us about using reporting guidelines checklists as evaluation tools — for what they have not been created. I saw no discussion about this in your paper.
We have now discussed this point in the paragraph before the limitations of the study.

10) Discussion. The discussion of the paper is short, and it lacks several points pointed above. Mainly, it does not discuss the limitations and potential bias of this study. Discussing limitations is a requirement from F1000Research journal.
We have now included such a section in the Discussion section.

11) You mention you did analyse qualitative information using NVivo. How did you do it?
We have now provided further information on how we did the qualitative analysis with NVivo (“Data analysis” section).

12) There are at least eight items from the CHERRIES checklist that I could not see in your paper, and you might want to report, even if in a separate (or supplementary) box, from data protection, preventing multiple entries, to randomisation of items, and others.
We have now included the information corresponding to the missing items throughout the whole Methods section.

13) From Box 1, I do not quite get the difference between Interventions 1 and 2. It seems that both ask authors to submit a populated/completed RG checklist. Is it that #1 asks for page numbers and #2 does not?
#1 asks for page numbers and #2 for text from the manuscript. Some people may think that if you nudge authors to have to copy-paste text into the checklist they may tend to take it more seriously than if you just ask them to put a page number. We have slightly modified their wording in Box 1 to make it clearer.

14) In my opinion, the sentences from “To contact editors...” to “improve adherence to RGs” belong more to the “Participants” section than to the “Procedure”. Especially the first sentence. Also “Participants could suggest further editors that they considered could contribute to the survey” is something that I would consider as a snowballing or recruitment method, not the survey procedure per se. So, I would suggest putting everything related to recruitment together in one section, to make it easier for your reader (who might be interested in undertaking a similar survey) to find information on such a critical issue as recruitment. Also: how could participants suggest? Was it a specific question in the SurveyMonkey form?
We have restructured that part of the manuscript and divided it into “Recruitment” and “Survey administration”. We have also provided the information requested.

15) Were participants given an estimation of the time required to answer the survey before they began?
We have now included this information (“Survey administration” subsection).

Minor: about citations
16) Reference number 1 (EQUATOR Network website) is not a proper, complete reference. It points to the home page of the website, not to a document, and as so, it does not show from where the authors collected that information (the reader cannot find them in the home page). The correct way of citing would be adding the full URL. A suggestion: EQUATOR Network. What is a reporting guideline? Available at:
http://www.equator-network.org/about-us/what-is-a-reporting-guideline/. Cited on 19/11/2019.
Done.

17) Something similar happens to reference number 13. Author and title are missing. It would help readers if it could be typed like this:
Schoter S. Evidence-based publishing. BMJ. Available at: https://www.bmj.com/about-bmj/evidence-based-publishing. Cited on 19/11/2019.
Done.

18) Reference 14 also lacks authors and titles. References 13, 14, 16, 21 and 22 lack a lot of elements necessary for readers to be able to find them. When working in a printed version (as I am now), you cannot see what the document is, where it is published or by whom — these are available only by those reading online and able to “click” on the links.
Done.

19) I would change the sentence:
“Raw survey results are given as Underlying data20.”
To something clearer and that does not require the reader to go to the reference list to understand what it is.
“Table XX in the supplementary file X20 shows the responses from all 24 participants.”
Done.

20) Has the study by Hawwash et al. (reference 25) been published yet? (The authors cited the protocol only). Do the authors plan to cite other electronic tools that are out there?
We have now cited the results paper by Hawwash et al., which got published on 6 November. Regarding the electronic tools, we have already cited COBWEB and Statreviewer. We did not cite Penelope as it does not explicitly check RG requirements – it just helps authors submit the relevant RG checklist.
We thank you for your time and detailed comments on the manuscript. Please find below a point by point response to them:

Major concerns: methodology

1) Sample characteristics. Although the authors should be praised for the idea of asking editors about interventions to improve reporting, I have a concern about what editors should be surveyed. If one wants to explore the perceived ease of the implementation of one intervention, is it adequate to choose a population of editors whom we know are already “knowledgeable about the topic”?
If these editors are aware of the reporting quality crisis and the reporting guidelines in general, wouldn’t these be more prone to adopt interventions that aim to improve the quality of publications? Therefore, aren’t these the people who would say “yes” to any intervention proposed?
Should the authors not interview editors who have never heard about reporting guidelines too? Wouldn’t this heterogeneous sample be more representative of the scenario of difficulties — or the ease — of implementing reporting guidelines checks in the editorial process? Are the study results only applicable to journals that already endorse or require reporting guidelines?
I would like to see comments about this limitation in the discussion if the authors agree with this or even if they do not.

Firstly, we have now included some more information in the “Participants” section to make it clear what kind of participants we were aiming to reach. Secondly, we have discussed in the Discussion section the other points you mention.

2) Sample size. My second most important concern is about sample size. In the Methods section, the authors do not explain how they planned the sample size and based on what assumptions. 24% seems to be a very low response rate. Do the authors believe 24 is enough? This limitation should be discussed.
We have now discussed these points in the Discussion section.

3) Effectiveness. When you asked your participants about their opinions on “how effective” something would be, what was the definition of effectiveness? Would it be their perception about the intervention improving the adherence to reporting guidelines checklists? Was it a more subjective view of “manuscript quality”?
Thanks to your comment, we have just realised that we only mentioned that we were talking about “effectiveness at improving adherence to RGs” in the methods section, but not in the abstract and introduction. We have now clarified that in the abstract and introduction, and also in some parts of the methods, results and discussion to remind the readers that we refer to effectiveness at improving adherence to RGs.

4) Statreviewer. The authors mention in the text, in Methods:
“Based on feedback from the pilot we decided not to include the intervention “Implementation of the automatic tool Statreviewer14” since participants were not aware of this software and stated that their perceptions would strongly depend on details about how it operates which are not publicly available.”
However, on page 7, Results, they state:
“The implementation of reading tools that automatically assess adherence to RGs, such as Statreviewer14, were seen as potentially interesting interventions.”
After all, was this intervention Statreviewer evaluated (as in Results) or was it not (as in Methods)? Do you mean they mentioned other tools, such as Penelope, GoodReports, Cobweb or others? Or were they referring to Statreviewer?
We have clarified in the results section (“Further interventions and incentives for authors and journals”) that Statreviewer was not included in the survey but mentioned by some participants when asked about further potentially effective interventions.

Major concerns: reporting
5) Journals. You interviewed 24 editors from 20 journals. What are these journals? I did not see a list of their names, and I believe it is important.
The 20 journals represented are now listed in Table 1. We have also improved the description of the sample in the first paragraph of the Results section.

6) Figure 1. Figure 1 is difficult to read/interpret. I like the idea of having the two outcomes (ease of implementation and effectiveness) in one figure very much so that one can compare the two. However, the bar chart is not clear at all, nor easy to interpret. Colours do not help. The authors should consider building a figure with plot points (which would be easy, given the small sample) showing where each participant sits within the Likert scale — not bars.
We have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.

Another suggestion for Figure 1 (here, just a suggestion) is to add something to remind the readers what the interventions are. As Box 1 is far from the Figure, it is difficult to remember what each intervention is. It would be useful to have “nicknames”, for example:
1. Page numbers
2. Checklist
3. Highlight
4. Subheadings
5. COBWEB
6. Use RG
7. Check RG
8. Staff
9. Training
We have incorporated similar nicknames to help readers remember the interventions.

7) Peer reviewers’ checks. The abstract is, in general, well written and clear. However, I got intrigued by this sentence:
“Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and could decrease the overall quality of reviews.”
I was surprised that peer reviewers checking reporting guidelines adherence would decrease the overall quality of the paper. Then I reread it in the paper, but here put in a complete sentence, that makes sense:
“Participants were concerned that the quality of peer review could be compromised as reviewers should focus on the manuscript’s content and not on the reporting issues.”
Meaning, I suppose, that peer reviewers would focus on checking reporting guidelines and not on their “expert view” of the study. I get it, but it is not very clear, neither in the text, nor in the abstract, and I invite the authors to rewrite both. Especially because “reporting guidelines” are about content too. They point out the content elements that should not be absent from papers, so using the word “content” in the manuscript text is not really appropriate. Please, re-evaluate.
We completely agree with the point you raise. We have re-written this point in the Abstract, Results and Conclusions. “Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and were concerned that the quality of peer review could be compromised. Reviewers are generally not expected to focus on reporting issues but on providing an expert view on the importance, novelty, and relevance of the manuscript.”

8) As I said above, the abstract is good, and there is room for growth (you have now 232 words). However, I could not find the conclusions in the abstract helpful. Could you please inform better what you did conclude? What “points raised in the survey” should be considered?
We have re-written and expanded the Conclusions in the abstract. The abstract is now 295 words.

9) Checklists as forms. ...“assessing completeness of reporting is a subjective task”. This is one of the most important sentences in this manuscript, it tells us about using reporting guidelines checklists as evaluation tools — for what they have not been created. I saw no discussion about this in your paper.
We have now discussed this point in the paragraph before the limitations of the study.

10) Discussion. The discussion of the paper is short, and it lacks several points pointed above. Mainly, it does not discuss the limitations and potential bias of this study. Discussing limitations is a requirement from F1000Research journal.
We have now included such a section in the Discussion section.

11) You mention you did analyse qualitative information using NVivo. How did you do it?
We have now provided further information on how we did the qualitative analysis with NVivo (“Data analysis” section).

12) There are at least eight items from the CHERRIES checklist that I could not see in your paper, and you might want to report, even if in a separate (or supplementary) box, from data protection, preventing multiple entries, to randomisation of items, and others.
We have now included the information corresponding to the missing items throughout the whole Methods section.

13) From Box 1, I do not quite get the difference between Interventions 1 and 2. It seems that both ask authors to submit a populated/completed RG checklist. Is it that #1 asks for page numbers and #2 does not?
#1 asks for page numbers and #2 for text from the manuscript. Some people may think that if you nudge authors to have to copy-paste text into the checklist they may tend to take it more seriously than if you just ask them to put a page number. We have slightly modified their wording in Box 1 to make it clearer.

14) In my opinion, the sentences from “To contact editors...” to “improve adherence to RGs” belong more to the “Participants” section than to the “Procedure”. Especially the first sentence. Also “Participants could suggest further editors that they considered could contribute to the survey” is something that I would consider as a snowballing or recruitment method, not the survey procedure per se. So, I would suggest putting everything related to recruitment together in one section, to make it easier for your reader (who might be interested in undertaking a similar survey) to find information on such a critical issue as recruitment. Also: how could participants suggest? Was it a specific question in the SurveyMonkey form?
We have restructured that part of the manuscript and divided it into “Recruitment” and “Survey administration”. We have also provided the information requested.

15) Were participants given an estimation of the time required to answer the survey before they began?
We have now included this information (“Survey administration” subsection).

Minor: about citations
16) Reference number 1 (EQUATOR Network website) is not a proper, complete reference. It points to the home page of the website, not to a document, and as so, it does not show from where the authors collected that information (the reader cannot find them in the home page). The correct way of citing would be adding the full URL. A suggestion: EQUATOR Network. What is a reporting guideline? Available at:
http://www.equator-network.org/about-us/what-is-a-reporting-guideline/. Cited on 19/11/2019.
Done.

17) Something similar happens to reference number 13. Author and title are missing. It would help readers if it could be typed like this:
Schoter S. Evidence-based publishing. BMJ. Available at: https://www.bmj.com/about-bmj/evidence-based-publishing. Cited on 19/11/2019.
Done.

18) Reference 14 also lacks authors and titles. References 13, 14, 16, 21 and 22 lack a lot of elements necessary for readers to be able to find them. When working in a printed version (as I am now), you cannot see what the document is, where it is published or by whom — these are available only by those reading online and able to “click” on the links.
Done.

19) I would change the sentence:
“Raw survey results are given as Underlying data20.”
To something clearer and that does not require the reader to go to the reference list to understand what it is.
“Table XX in the supplementary file X20 shows the responses from all 24 participants.”
Done.

20) Has the study by Hawwash et al. (reference 25) been published yet? (The authors cited the protocol only). Do the authors plan to cite other electronic tools that are out there?
We have now cited the results paper by Hawwash et al., which got published on 6 November. Regarding the electronic tools, we have already cited COBWEB and Statreviewer. We did not cite Penelope as it does not explicitly check RG requirements – it just helps authors submit the relevant RG checklist.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 12 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

12 Dec 2019

Author Response

We thank you for your time and detailed comments on the manuscript. Please find below a point by point response to them:

Major concerns: methodology

1) Sample characteristics. Although the authors should ... Continue reading We thank you for your time and detailed comments on the manuscript. Please find below a point by point response to them:

Major concerns: methodology

1) Sample characteristics. Although the authors should be praised for the idea of asking editors about interventions to improve reporting, I have a concern about what editors should be surveyed. If one wants to explore the perceived ease of the implementation of one intervention, is it adequate to choose a population of editors whom we know are already “knowledgeable about the topic”?
If these editors are aware of the reporting quality crisis and the reporting guidelines in general, wouldn’t these be more prone to adopt interventions that aim to improve the quality of publications? Therefore, aren’t these the people who would say “yes” to any intervention proposed?
Should the authors not interview editors who have never heard about reporting guidelines too? Wouldn’t this heterogeneous sample be more representative of the scenario of difficulties — or the ease — of implementing reporting guidelines checks in the editorial process? Are the study results only applicable to journals that already endorse or require reporting guidelines?
I would like to see comments about this limitation in the discussion if the authors agree with this or even if they do not.

Firstly, we have now included some more information in the “Participants” section to make it clear what kind of participants we were aiming to reach. Secondly, we have discussed in the Discussion section the other points you mention.

2) Sample size. My second most important concern is about sample size. In the Methods section, the authors do not explain how they planned the sample size and based on what assumptions. 24% seems to be a very low response rate. Do the authors believe 24 is enough? This limitation should be discussed.
We have now discussed these points in the Discussion section.

3) Effectiveness. When you asked your participants about their opinions on “how effective” something would be, what was the definition of effectiveness? Would it be their perception about the intervention improving the adherence to reporting guidelines checklists? Was it a more subjective view of “manuscript quality”?
Thanks to your comment, we have just realised that we only mentioned that we were talking about “effectiveness at improving adherence to RGs” in the methods section, but not in the abstract and introduction. We have now clarified that in the abstract and introduction, and also in some parts of the methods, results and discussion to remind the readers that we refer to effectiveness at improving adherence to RGs.

4) Statreviewer. The authors mention in the text, in Methods:
“Based on feedback from the pilot we decided not to include the intervention “Implementation of the automatic tool Statreviewer14” since participants were not aware of this software and stated that their perceptions would strongly depend on details about how it operates which are not publicly available.”
However, on page 7, Results, they state:
“The implementation of reading tools that automatically assess adherence to RGs, such as Statreviewer14, were seen as potentially interesting interventions.”
After all, was this intervention Statreviewer evaluated (as in Results) or was it not (as in Methods)? Do you mean they mentioned other tools, such as Penelope, GoodReports, Cobweb or others? Or were they referring to Statreviewer?
We have clarified in the results section (“Further interventions and incentives for authors and journals”) that Statreviewer was not included in the survey but mentioned by some participants when asked about further potentially effective interventions.

Major concerns: reporting
5) Journals. You interviewed 24 editors from 20 journals. What are these journals? I did not see a list of their names, and I believe it is important.
The 20 journals represented are now listed in Table 1. We have also improved the description of the sample in the first paragraph of the Results section.

6) Figure 1. Figure 1 is difficult to read/interpret. I like the idea of having the two outcomes (ease of implementation and effectiveness) in one figure very much so that one can compare the two. However, the bar chart is not clear at all, nor easy to interpret. Colours do not help. The authors should consider building a figure with plot points (which would be easy, given the small sample) showing where each participant sits within the Likert scale — not bars.
We have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.

Another suggestion for Figure 1 (here, just a suggestion) is to add something to remind the readers what the interventions are. As Box 1 is far from the Figure, it is difficult to remember what each intervention is. It would be useful to have “nicknames”, for example:
1. Page numbers
2. Checklist
3. Highlight
4. Subheadings
5. COBWEB
6. Use RG
7. Check RG
8. Staff
9. Training
We have incorporated similar nicknames to help readers remember the interventions.

7) Peer reviewers’ checks. The abstract is, in general, well written and clear. However, I got intrigued by this sentence:
“Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and could decrease the overall quality of reviews.”
I was surprised that peer reviewers checking reporting guidelines adherence would decrease the overall quality of the paper. Then I reread it in the paper, but here put in a complete sentence, that makes sense:
“Participants were concerned that the quality of peer review could be compromised as reviewers should focus on the manuscript’s content and not on the reporting issues.”
Meaning, I suppose, that peer reviewers would focus on checking reporting guidelines and not on their “expert view” of the study. I get it, but it is not very clear, neither in the text, nor in the abstract, and I invite the authors to rewrite both. Especially because “reporting guidelines” are about content too. They point out the content elements that should not be absent from papers, so using the word “content” in the manuscript text is not really appropriate. Please, re-evaluate.
We completely agree with the point you raise. We have re-written this point in the Abstract, Results and Conclusions. “Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and were concerned that the quality of peer review could be compromised. Reviewers are generally not expected to focus on reporting issues but on providing an expert view on the importance, novelty, and relevance of the manuscript.”

8) As I said above, the abstract is good, and there is room for growth (you have now 232 words). However, I could not find the conclusions in the abstract helpful. Could you please inform better what you did conclude? What “points raised in the survey” should be considered?
We have re-written and expanded the Conclusions in the abstract. The abstract is now 295 words.

9) Checklists as forms. ...“assessing completeness of reporting is a subjective task”. This is one of the most important sentences in this manuscript, it tells us about using reporting guidelines checklists as evaluation tools — for what they have not been created. I saw no discussion about this in your paper.
We have now discussed this point in the paragraph before the limitations of the study.

10) Discussion. The discussion of the paper is short, and it lacks several points pointed above. Mainly, it does not discuss the limitations and potential bias of this study. Discussing limitations is a requirement from F1000Research journal.
We have now included such a section in the Discussion section.

11) You mention you did analyse qualitative information using NVivo. How did you do it?
We have now provided further information on how we did the qualitative analysis with NVivo (“Data analysis” section).

12) There are at least eight items from the CHERRIES checklist that I could not see in your paper, and you might want to report, even if in a separate (or supplementary) box, from data protection, preventing multiple entries, to randomisation of items, and others.
We have now included the information corresponding to the missing items throughout the whole Methods section.

13) From Box 1, I do not quite get the difference between Interventions 1 and 2. It seems that both ask authors to submit a populated/completed RG checklist. Is it that #1 asks for page numbers and #2 does not?
#1 asks for page numbers and #2 for text from the manuscript. Some people may think that if you nudge authors to have to copy-paste text into the checklist they may tend to take it more seriously than if you just ask them to put a page number. We have slightly modified their wording in Box 1 to make it clearer.

14) In my opinion, the sentences from “To contact editors...” to “improve adherence to RGs” belong more to the “Participants” section than to the “Procedure”. Especially the first sentence. Also “Participants could suggest further editors that they considered could contribute to the survey” is something that I would consider as a snowballing or recruitment method, not the survey procedure per se. So, I would suggest putting everything related to recruitment together in one section, to make it easier for your reader (who might be interested in undertaking a similar survey) to find information on such a critical issue as recruitment. Also: how could participants suggest? Was it a specific question in the SurveyMonkey form?
We have restructured that part of the manuscript and divided it into “Recruitment” and “Survey administration”. We have also provided the information requested.

15) Were participants given an estimation of the time required to answer the survey before they began?
We have now included this information (“Survey administration” subsection).

Minor: about citations
16) Reference number 1 (EQUATOR Network website) is not a proper, complete reference. It points to the home page of the website, not to a document, and as so, it does not show from where the authors collected that information (the reader cannot find them in the home page). The correct way of citing would be adding the full URL. A suggestion: EQUATOR Network. What is a reporting guideline? Available at:
http://www.equator-network.org/about-us/what-is-a-reporting-guideline/. Cited on 19/11/2019.
Done.

17) Something similar happens to reference number 13. Author and title are missing. It would help readers if it could be typed like this:
Schoter S. Evidence-based publishing. BMJ. Available at: https://www.bmj.com/about-bmj/evidence-based-publishing. Cited on 19/11/2019.
Done.

18) Reference 14 also lacks authors and titles. References 13, 14, 16, 21 and 22 lack a lot of elements necessary for readers to be able to find them. When working in a printed version (as I am now), you cannot see what the document is, where it is published or by whom — these are available only by those reading online and able to “click” on the links.
Done.

19) I would change the sentence:
“Raw survey results are given as Underlying data20.”
To something clearer and that does not require the reader to go to the reference list to understand what it is.
“Table XX in the supplementary file X20 shows the responses from all 24 participants.”
Done.

20) Has the study by Hawwash et al. (reference 25) been published yet? (The authors cited the protocol only). Do the authors plan to cite other electronic tools that are out there?
We have now cited the results paper by Hawwash et al., which got published on 6 November. Regarding the electronic tools, we have already cited COBWEB and Statreviewer. We did not cite Penelope as it does not explicitly check RG requirements – it just helps authors submit the relevant RG checklist.
We thank you for your time and detailed comments on the manuscript. Please find below a point by point response to them:

Major concerns: methodology

1) Sample characteristics. Although the authors should be praised for the idea of asking editors about interventions to improve reporting, I have a concern about what editors should be surveyed. If one wants to explore the perceived ease of the implementation of one intervention, is it adequate to choose a population of editors whom we know are already “knowledgeable about the topic”?
If these editors are aware of the reporting quality crisis and the reporting guidelines in general, wouldn’t these be more prone to adopt interventions that aim to improve the quality of publications? Therefore, aren’t these the people who would say “yes” to any intervention proposed?
Should the authors not interview editors who have never heard about reporting guidelines too? Wouldn’t this heterogeneous sample be more representative of the scenario of difficulties — or the ease — of implementing reporting guidelines checks in the editorial process? Are the study results only applicable to journals that already endorse or require reporting guidelines?
I would like to see comments about this limitation in the discussion if the authors agree with this or even if they do not.

Firstly, we have now included some more information in the “Participants” section to make it clear what kind of participants we were aiming to reach. Secondly, we have discussed in the Discussion section the other points you mention.

2) Sample size. My second most important concern is about sample size. In the Methods section, the authors do not explain how they planned the sample size and based on what assumptions. 24% seems to be a very low response rate. Do the authors believe 24 is enough? This limitation should be discussed.
We have now discussed these points in the Discussion section.

3) Effectiveness. When you asked your participants about their opinions on “how effective” something would be, what was the definition of effectiveness? Would it be their perception about the intervention improving the adherence to reporting guidelines checklists? Was it a more subjective view of “manuscript quality”?
Thanks to your comment, we have just realised that we only mentioned that we were talking about “effectiveness at improving adherence to RGs” in the methods section, but not in the abstract and introduction. We have now clarified that in the abstract and introduction, and also in some parts of the methods, results and discussion to remind the readers that we refer to effectiveness at improving adherence to RGs.

4) Statreviewer. The authors mention in the text, in Methods:
“Based on feedback from the pilot we decided not to include the intervention “Implementation of the automatic tool Statreviewer14” since participants were not aware of this software and stated that their perceptions would strongly depend on details about how it operates which are not publicly available.”
However, on page 7, Results, they state:
“The implementation of reading tools that automatically assess adherence to RGs, such as Statreviewer14, were seen as potentially interesting interventions.”
After all, was this intervention Statreviewer evaluated (as in Results) or was it not (as in Methods)? Do you mean they mentioned other tools, such as Penelope, GoodReports, Cobweb or others? Or were they referring to Statreviewer?
We have clarified in the results section (“Further interventions and incentives for authors and journals”) that Statreviewer was not included in the survey but mentioned by some participants when asked about further potentially effective interventions.

Major concerns: reporting
5) Journals. You interviewed 24 editors from 20 journals. What are these journals? I did not see a list of their names, and I believe it is important.
The 20 journals represented are now listed in Table 1. We have also improved the description of the sample in the first paragraph of the Results section.

6) Figure 1. Figure 1 is difficult to read/interpret. I like the idea of having the two outcomes (ease of implementation and effectiveness) in one figure very much so that one can compare the two. However, the bar chart is not clear at all, nor easy to interpret. Colours do not help. The authors should consider building a figure with plot points (which would be easy, given the small sample) showing where each participant sits within the Likert scale — not bars.
We have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.

Another suggestion for Figure 1 (here, just a suggestion) is to add something to remind the readers what the interventions are. As Box 1 is far from the Figure, it is difficult to remember what each intervention is. It would be useful to have “nicknames”, for example:
1. Page numbers
2. Checklist
3. Highlight
4. Subheadings
5. COBWEB
6. Use RG
7. Check RG
8. Staff
9. Training
We have incorporated similar nicknames to help readers remember the interventions.

7) Peer reviewers’ checks. The abstract is, in general, well written and clear. However, I got intrigued by this sentence:
“Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and could decrease the overall quality of reviews.”
I was surprised that peer reviewers checking reporting guidelines adherence would decrease the overall quality of the paper. Then I reread it in the paper, but here put in a complete sentence, that makes sense:
“Participants were concerned that the quality of peer review could be compromised as reviewers should focus on the manuscript’s content and not on the reporting issues.”
Meaning, I suppose, that peer reviewers would focus on checking reporting guidelines and not on their “expert view” of the study. I get it, but it is not very clear, neither in the text, nor in the abstract, and I invite the authors to rewrite both. Especially because “reporting guidelines” are about content too. They point out the content elements that should not be absent from papers, so using the word “content” in the manuscript text is not really appropriate. Please, re-evaluate.
We completely agree with the point you raise. We have re-written this point in the Abstract, Results and Conclusions. “Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and were concerned that the quality of peer review could be compromised. Reviewers are generally not expected to focus on reporting issues but on providing an expert view on the importance, novelty, and relevance of the manuscript.”

8) As I said above, the abstract is good, and there is room for growth (you have now 232 words). However, I could not find the conclusions in the abstract helpful. Could you please inform better what you did conclude? What “points raised in the survey” should be considered?
We have re-written and expanded the Conclusions in the abstract. The abstract is now 295 words.

9) Checklists as forms. ...“assessing completeness of reporting is a subjective task”. This is one of the most important sentences in this manuscript, it tells us about using reporting guidelines checklists as evaluation tools — for what they have not been created. I saw no discussion about this in your paper.
We have now discussed this point in the paragraph before the limitations of the study.

10) Discussion. The discussion of the paper is short, and it lacks several points pointed above. Mainly, it does not discuss the limitations and potential bias of this study. Discussing limitations is a requirement from F1000Research journal.
We have now included such a section in the Discussion section.

11) You mention you did analyse qualitative information using NVivo. How did you do it?
We have now provided further information on how we did the qualitative analysis with NVivo (“Data analysis” section).

12) There are at least eight items from the CHERRIES checklist that I could not see in your paper, and you might want to report, even if in a separate (or supplementary) box, from data protection, preventing multiple entries, to randomisation of items, and others.
We have now included the information corresponding to the missing items throughout the whole Methods section.

13) From Box 1, I do not quite get the difference between Interventions 1 and 2. It seems that both ask authors to submit a populated/completed RG checklist. Is it that #1 asks for page numbers and #2 does not?
#1 asks for page numbers and #2 for text from the manuscript. Some people may think that if you nudge authors to have to copy-paste text into the checklist they may tend to take it more seriously than if you just ask them to put a page number. We have slightly modified their wording in Box 1 to make it clearer.

14) In my opinion, the sentences from “To contact editors...” to “improve adherence to RGs” belong more to the “Participants” section than to the “Procedure”. Especially the first sentence. Also “Participants could suggest further editors that they considered could contribute to the survey” is something that I would consider as a snowballing or recruitment method, not the survey procedure per se. So, I would suggest putting everything related to recruitment together in one section, to make it easier for your reader (who might be interested in undertaking a similar survey) to find information on such a critical issue as recruitment. Also: how could participants suggest? Was it a specific question in the SurveyMonkey form?
We have restructured that part of the manuscript and divided it into “Recruitment” and “Survey administration”. We have also provided the information requested.

15) Were participants given an estimation of the time required to answer the survey before they began?
We have now included this information (“Survey administration” subsection).

Minor: about citations
16) Reference number 1 (EQUATOR Network website) is not a proper, complete reference. It points to the home page of the website, not to a document, and as so, it does not show from where the authors collected that information (the reader cannot find them in the home page). The correct way of citing would be adding the full URL. A suggestion: EQUATOR Network. What is a reporting guideline? Available at:
http://www.equator-network.org/about-us/what-is-a-reporting-guideline/. Cited on 19/11/2019.
Done.

17) Something similar happens to reference number 13. Author and title are missing. It would help readers if it could be typed like this:
Schoter S. Evidence-based publishing. BMJ. Available at: https://www.bmj.com/about-bmj/evidence-based-publishing. Cited on 19/11/2019.
Done.

18) Reference 14 also lacks authors and titles. References 13, 14, 16, 21 and 22 lack a lot of elements necessary for readers to be able to find them. When working in a printed version (as I am now), you cannot see what the document is, where it is published or by whom — these are available only by those reading online and able to “click” on the links.
Done.

19) I would change the sentence:
“Raw survey results are given as Underlying data20.”
To something clearer and that does not require the reader to go to the reference list to understand what it is.
“Table XX in the supplementary file X20 shows the responses from all 24 participants.”
Done.

20) Has the study by Hawwash et al. (reference 25) been published yet? (The authors cited the protocol only). Do the authors plan to cite other electronic tools that are out there?
We have now cited the results paper by Hawwash et al., which got published on 6 November. Regarding the electronic tools, we have already cited COBWEB and Statreviewer. We did not cite Penelope as it does not explicitly check RG requirements – it just helps authors submit the relevant RG checklist.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 20 Nov 2019

Gerard Urrutia, Iberoamerican Cochrane Center, Sant Pau Biomedical Research Institute, Barcelona, Spain

Approved

https://doi.org/10.5256/f1000research.22597.r55889

This is an interesting and original study that deepens the knowledge about interventions aimed at improving adherence to reporting guidelines by medical journals, an important issue. The study focuses on the perspective of the biomedical editors and provides valuable information on their perceptions of various interventions that have been proposed, identifies barriers and facilitators and provides possible solutions.

However, the study includes a limited sample of participants (24 editors belonging to 20 medical journals), which limits the scope of the conclusions. Although the study, according to its stated objectives, does not intend to offer a representative view of the editors of all medical journals, the limited number of participants as well as their representativeness (the sources from which they were identified and selected allows us to think that it is a highly selected sample of publishers with previous experience and a potential motivation in the subject), is not mentioned in the discussion. It is noteworthy that, despite this potentially "favourable" sample, the response rate is very low (24%). What would have been the participation rate and the responses if a larger and more representative sample had been selected? Although this was not an objective of the study, it would deserve some reflection in the discussion.

It would also be useful to provide details on the origin of the participants in the survey according to the sources of selection, which would improve the description of the sample.

The results presented both in the section on current practices as well as in perceptions and in barriers and facilitators are consistent, clear and practical. The analysis is descriptive and does not pose difficulties.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Systematic reviews

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Author Response 12 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

12 Dec 2019

Author Response

We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

However, the study includes a limited sample of ... Continue reading We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

However, the study includes a limited sample of participants (24 editors belonging to 20 medical journals), which limits the scope of the conclusions. Although the study, according to its stated objectives, does not intend to offer a representative view of the editors of all medical journals, the limited number of participants as well as their representativeness (the sources from which they were identified and selected allows us to think that it is a highly selected sample of publishers with previous experience and a potential motivation in the subject), is not mentioned in the discussion.

We have now discussed these points in the Discussion section.

It is noteworthy that, despite this potentially "favourable" sample, the response rate is very low (24%). What would have been the participation rate and the responses if a larger and more representative sample had been selected? Although this was not an objective of the study, it would deserve some reflection in the discussion.

We have now included some comments about the response rate in a new paragraph on Limitations (Discussion section).

It would also be useful to provide details on the origin of the participants in the survey according to the sources of selection, which would improve the description of the sample.

We have now added that information to the first paragraph of the Results section.
We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

However, the study includes a limited sample of participants (24 editors belonging to 20 medical journals), which limits the scope of the conclusions. Although the study, according to its stated objectives, does not intend to offer a representative view of the editors of all medical journals, the limited number of participants as well as their representativeness (the sources from which they were identified and selected allows us to think that it is a highly selected sample of publishers with previous experience and a potential motivation in the subject), is not mentioned in the discussion.

We have now discussed these points in the Discussion section.

It is noteworthy that, despite this potentially "favourable" sample, the response rate is very low (24%). What would have been the participation rate and the responses if a larger and more representative sample had been selected? Although this was not an objective of the study, it would deserve some reflection in the discussion.

We have now included some comments about the response rate in a new paragraph on Limitations (Discussion section).

It would also be useful to provide details on the origin of the participants in the survey according to the sources of selection, which would improve the description of the sample.

We have now added that information to the first paragraph of the Results section.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 12 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

12 Dec 2019

Author Response

We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

However, the study includes a limited sample of ... Continue reading We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

However, the study includes a limited sample of participants (24 editors belonging to 20 medical journals), which limits the scope of the conclusions. Although the study, according to its stated objectives, does not intend to offer a representative view of the editors of all medical journals, the limited number of participants as well as their representativeness (the sources from which they were identified and selected allows us to think that it is a highly selected sample of publishers with previous experience and a potential motivation in the subject), is not mentioned in the discussion.

We have now discussed these points in the Discussion section.

It is noteworthy that, despite this potentially "favourable" sample, the response rate is very low (24%). What would have been the participation rate and the responses if a larger and more representative sample had been selected? Although this was not an objective of the study, it would deserve some reflection in the discussion.

We have now included some comments about the response rate in a new paragraph on Limitations (Discussion section).

It would also be useful to provide details on the origin of the participants in the survey according to the sources of selection, which would improve the description of the sample.

We have now added that information to the first paragraph of the Results section.
We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

However, the study includes a limited sample of participants (24 editors belonging to 20 medical journals), which limits the scope of the conclusions. Although the study, according to its stated objectives, does not intend to offer a representative view of the editors of all medical journals, the limited number of participants as well as their representativeness (the sources from which they were identified and selected allows us to think that it is a highly selected sample of publishers with previous experience and a potential motivation in the subject), is not mentioned in the discussion.

We have now discussed these points in the Discussion section.

It is noteworthy that, despite this potentially "favourable" sample, the response rate is very low (24%). What would have been the participation rate and the responses if a larger and more representative sample had been selected? Although this was not an objective of the study, it would deserve some reflection in the discussion.

We have now included some comments about the response rate in a new paragraph on Limitations (Discussion section).

It would also be useful to provide details on the origin of the participants in the survey according to the sources of selection, which would improve the description of the sample.

We have now added that information to the first paragraph of the Results section.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 11 Nov 2019

Dennis W. Lendrem, Translational and Clinical Research Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK; NIHR Newcastle Biomedical Research Centre, Newcastle University and Newcastle upon Tyne Hospitals NHS Trust, Newcastle upon Tyne, UK

Approved with Reservations

https://doi.org/10.5256/f1000research.22597.r55890

This paper reports on the views of a sample of biomedical editors on editorial interventions to improve adherence to reporting guidelines. The paper is clear, transparent, and well documented.

A limitation of the study is that the results are based upon a survey return rate of just 24%. The returns are likely biased favouring editors with stronger views on reporting guidelines. In addition, it is not clear whether all ten of the top-ten journals were represented, or whether one or more journals dominate the response rate. In addition it is not completely clear how many participants came from the MiRoR Network, or journals previously publishing studies on Reporting Guidelines.

Figure 1 is hard work to interpret and might be presented more usefully as two separate graphs. In addition, the colour scheme could be changed to highlight those interventions considered "Very easy" or "Moderately easy" to implement, and those ranked as "Very effective" or "Moderately effective". (Note that the proportions of each score could still be retained.)

However, this is a useful addition to the literature offering valuable insight into some views on guidelines and candidate interventions promoting closer adherence to guidelines. The paper would be improved by a paragraph on Limitations of the Study.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: statistics, modelling, validation, decision making

CITE

Report a concern

Author Response 12 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

12 Dec 2019

Author Response

We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

A limitation of the study is that the results are ... Continue reading We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

A limitation of the study is that the results are based upon a survey return rate of just 24%. The returns are likely biased favouring editors with stronger views on reporting guidelines.

We have now reflected on this point in the Discussion section.

In addition, it is not clear whether all ten of the top-ten journals were represented, or whether one or more journals dominate the response rate. In addition it is not completely clear how many participants came from the MiRoR Network, or journals previously publishing studies on Reporting Guidelines.

We have now added that information to the first paragraph of the Results section.

Figure 1 is hard work to interpret and might be presented more usefully as two separate graphs. In addition, the colour scheme could be changed to highlight those interventions considered "Very easy" or "Moderately easy" to implement, and those ranked as "Very effective" or "Moderately effective". (Note that the proportions of each score could still be retained.)

In order to make Figure 1 easier to interpret, we have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.
However, we prefer not to separate it into two graphs as we believe it is interesting for the readers to be able to compare the scores that each intervention obtains for each of the two outcomes.

The paper would be improved by a paragraph on Limitations of the Study.

We have now included a section on Limitations of the study in the Discussion section.
We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

A limitation of the study is that the results are based upon a survey return rate of just 24%. The returns are likely biased favouring editors with stronger views on reporting guidelines.

We have now reflected on this point in the Discussion section.

In addition, it is not clear whether all ten of the top-ten journals were represented, or whether one or more journals dominate the response rate. In addition it is not completely clear how many participants came from the MiRoR Network, or journals previously publishing studies on Reporting Guidelines.

We have now added that information to the first paragraph of the Results section.

Figure 1 is hard work to interpret and might be presented more usefully as two separate graphs. In addition, the colour scheme could be changed to highlight those interventions considered "Very easy" or "Moderately easy" to implement, and those ranked as "Very effective" or "Moderately effective". (Note that the proportions of each score could still be retained.)

In order to make Figure 1 easier to interpret, we have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.
However, we prefer not to separate it into two graphs as we believe it is interesting for the readers to be able to compare the scores that each intervention obtains for each of the two outcomes.

The paper would be improved by a paragraph on Limitations of the Study.

We have now included a section on Limitations of the study in the Discussion section.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 12 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

12 Dec 2019

Author Response

We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

A limitation of the study is that the results are ... Continue reading We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

A limitation of the study is that the results are based upon a survey return rate of just 24%. The returns are likely biased favouring editors with stronger views on reporting guidelines.

We have now reflected on this point in the Discussion section.

In addition, it is not clear whether all ten of the top-ten journals were represented, or whether one or more journals dominate the response rate. In addition it is not completely clear how many participants came from the MiRoR Network, or journals previously publishing studies on Reporting Guidelines.

We have now added that information to the first paragraph of the Results section.

Figure 1 is hard work to interpret and might be presented more usefully as two separate graphs. In addition, the colour scheme could be changed to highlight those interventions considered "Very easy" or "Moderately easy" to implement, and those ranked as "Very effective" or "Moderately effective". (Note that the proportions of each score could still be retained.)

In order to make Figure 1 easier to interpret, we have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.
However, we prefer not to separate it into two graphs as we believe it is interesting for the readers to be able to compare the scores that each intervention obtains for each of the two outcomes.

The paper would be improved by a paragraph on Limitations of the Study.

We have now included a section on Limitations of the study in the Discussion section.
We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

A limitation of the study is that the results are based upon a survey return rate of just 24%. The returns are likely biased favouring editors with stronger views on reporting guidelines.

We have now reflected on this point in the Discussion section.

In addition, it is not clear whether all ten of the top-ten journals were represented, or whether one or more journals dominate the response rate. In addition it is not completely clear how many participants came from the MiRoR Network, or journals previously publishing studies on Reporting Guidelines.

We have now added that information to the first paragraph of the Results section.

Figure 1 is hard work to interpret and might be presented more usefully as two separate graphs. In addition, the colour scheme could be changed to highlight those interventions considered "Very easy" or "Moderately easy" to implement, and those ranked as "Very effective" or "Moderately effective". (Note that the proportions of each score could still be retained.)

In order to make Figure 1 easier to interpret, we have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.
However, we prefer not to separate it into two graphs as we believe it is interesting for the readers to be able to compare the scores that each intervention obtains for each of the two outcomes.

The paper would be improved by a paragraph on Limitations of the Study.

We have now included a section on Limitations of the study in the Discussion section.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 24 Sep 2019

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3
Version 3 (revision) 23 Dec 19		read	read
Version 2 (revision) 12 Dec 19	read		read
Version 1 24 Sep 19	read	read	read

Dennis W. Lendrem, Newcastle University, Newcastle upon Tyne, UK; Newcastle University and Newcastle upon Tyne Hospitals NHS Trust, Newcastle upon Tyne, UK
Gerard Urrutia, Sant Pau Biomedical Research Institute, Barcelona, Spain
Patricia Logullo, University of Oxford, Oxford, UK

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

14 Views

20 Jan 2020 | for Version 3

Gerard Urrutia, Iberoamerican Cochrane Center, Sant Pau Biomedical Research Institute, Barcelona, Spain

14 Views Cite this report Responses(0)

Approved

I’m happy with the revised version and my decision is to recommend approval as it is now.

Competing Interests

No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

21 Views

23 Dec 2019 | for Version 3

Patricia Logullo, EQUATOR Network, University of Oxford, Oxford, UK

21 Views Cite this report Responses(1)

Approved

I am very happy with this new version of the manuscript. It is clear, well-reported, and the conclusions are there. Thank you for taking the time to improve it! Thank you also for the opportunity to review it.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Methodology. Reporting quality. Research integrity.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Back to all reports

Reviewer Report

24 Views

16 Dec 2019 | for Version 2

Patricia Logullo, EQUATOR Network, University of Oxford, Oxford, UK

24 Views Cite this report Responses(1)

Approved With Reservations

Positive points

The paper improved a lot from the last version. As I said before, this is a much-needed survey by MiRoR team, and I hope it gets fully published and indexed as soon as possible.

The authors took care to respond to all the reviewers’ questions. However, there are still three points that were not adequately addressed, and I hope I can make some suggestions to help them solve these problems. I also found two additional problems that came up in this second version of the manuscript.

From the 20 points I commented in my last report, the authors adequately responded/solved the problems in the items 1, 3-5, 7, 9-17, 19 and 20. Thank you for that. Below you can find comments on the remaining items (I used the same numbers as before to make it easier to locate them) and the two additional ones.

2) Sample. The authors explained that they purposely searched for editors experienced with reporting guidelines. As much as I don’t agree with this approach, at least it was clearly stated now.

However, the explanation about the sample size as a study limitation still does not seem satisfactory. The authors added a comment in the discussion section attributing the sample size to a low response rate in surveys in general. But if the authors knew there was going to be a low response rate (and they cited very useful references for that), why did they stop recruiting? Why not try to approach other editors/journals? The authors should expand the explanation to respond to two questions:

What were the constraints in increasing the sample? Why did you stop recruiting?
Even choosing to use a “not representative sample”, can we trust that we should listen to the opinions of these 24 editors (a minority among the 99 invited) to say something is feasible or not, for example? What is the effect of this small sample size on the “take-home messages” of this study?

These points were not addressed in the paragraph about study limitations, and I think they should. Also: who were the editors who did not respond? Do you have information about them that you could show in a table? If you can find and show a pattern of response (for example, editors from high-visibility journals would have less interest in responding?), then we can understand better this low response rate.

6) Figure 1. Figure 1 improved a lot, thank you. Now it shows outliers and the types of interventions are clearer too (I wonder if you could put these “nicknames” in Box 1 too!).

However, it is still a bit difficult to read/interpret. I believe the problem is that you chose to show means when you probably should have used medians (more appropriate to scores and to data that is not normally distributed). I suggest the authors try to check for data distribution, try to generate this figure with means, and that they also describe this method in the Methods – Data analysis section.

8) The abstract improved, well done! However, the conclusions seem not to be conclusions at all, not based on your data.

The first sentence is your (and even mine!) opinion, but it not supported by your data. The second sentence tells about your aims, not your conclusions.

The same is valid for the conclusions at the end of the paper: they show valid comments/assumptions but nothing based on your data.

Useful conclusions should be linked to your aims (to survey editors’ perceptions) and read like “the editors think that”... or “the perceptions of the editors are...”.

18) References: you fixed most of them, but there are still references incomplete. Please review # 23, 24 and 28 and add the URLs.

Additional concerns:

Regarding Data Analysis: I noticed you added text in this section about “excluding one questionnaire that was terminated early”. That called my attention: is this one of the 24 (and then we have 23 responded questionnaires) or was it a 25^th questionnaire? Also: how did you approach missing data? By excluding the participants? Was there any other case?

Minor: Thank you for providing the names of the journals in Table 1. However, I suspect “Francis and Taylor” is not a journal name. Please, check (maybe there is a confusion with the publisher’s name “Taylor & Francis”, but what was the specific journal?).

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Methodology. Reporting quality. Research integrity.

Respond to this report

Responses (1)

Author Response

23 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

Thanks a lot, Patricia, for your extra comments and the time you took to make them. Please see below our response to them:

Minor comments
References: you fixed most of them, but there are still references incomplete. Please review # 23, 24 and 28 and add the URLs.
In our previous revised manuscript, we already had added the URLs for all Internet references. However, it seems that the copy-editors of F1000 did not explicitly show them and just include a link to them under “Reference source”.

Regarding Data Analysis: I noticed you added text in this section about “excluding one questionnaire that was terminated early”. That called my attention: is this one of the 24 (and then we have 23 responded questionnaires) or was it a 25th questionnaire? Also: how did you approach missing data? By excluding the participants? Was there any other case?
We excluded from analysis one questionnaire (the 25^th) where the participant just opened the survey (and therefore SurveyMonkey registered it) but left without answering any question. For this reason, we decided not to count it as a participant. There were no further cases like this. We have made that more explicit in the text (first paragraph of Data analysis, first paragraph of Results).

Minor: Thank you for providing the names of the journals in Table 1. However, I suspect “Francis and Taylor” is not a journal name. Please, check (maybe there is a confusion with the publisher’s name “Taylor & Francis”, but what was the specific journal?).
We have checked that that was a mistake. Thanks to this, we have realised that for one of the existing journals we had two editors instead of one. We have subsequently adjusted the sample description.

Major comments
2) Sample. The authors explained that they purposely searched for editors experienced with reporting guidelines. As much as I don’t agree with this approach, at least it was clearly stated now.
However, the explanation about the sample size as a study limitation still does not seem satisfactory. The authors added a comment in the discussion section attributing the sample size to a low response rate in surveys in general. But if the authors knew there was going to be a low response rate (and they cited very useful references for that), why did they stop recruiting? Why not try to approach other editors/journals? The authors should expand the explanation to respond to two questions:
What were the constraints in increasing the sample? Why did you stop recruiting?
Even choosing to use a “not representative sample”, can we trust that we should listen to the opinions of these 24 editors (a minority among the 99 invited) to say something is feasible or not, for example? What is the effect of this small sample size on the “take-home messages” of this study?
These points were not addressed in the paragraph about study limitations, and I think they should. Also: who were the editors who did not respond? Do you have information about them that you could show in a table? If you can find and show a pattern of response (for example, editors from high-visibility journals would have less interest in responding?), then we can understand better this low response rate.
We have added further justifications on whether we believed that sample size and response rates were adequate for the purpose of the survey: “Despite having the option to increase the sample size by contacting more editors at a lower level of hierarchy in the journals we targeted, we decided not to do it based on the response rate of the survey. That approach would have changed our sampling frame and we would potentially have had less experienced editors commenting. We took that decision as the purpose of the survey was to tap into the experience of those who had tried interventions or had shown interest in this area, instead of seeking a representative sample of editors.”
We found no pattern of response among the journals/editors surveyed – participation was pretty homogeneous and covered journals of a variety of impact factors and areas, as can be seen in Table 1.

6) Figure 1. Figure 1 improved a lot, thank you. Now it shows outliers and the types of interventions are clearer too (I wonder if you could put these “nicknames” in Box 1 too!).
However, it is still a bit difficult to read/interpret. I believe the problem is that you chose to show means when you probably should have used medians (more appropriate to scores and to data that is not normally distributed). I suggest the authors try to check for data distribution, try to generate this figure with means, and that they also describe this method in the Methods – Data analysis section.
We are now reporting the results for the ease of implementation/effectiveness in terms of medians and quartiles instead of means and standard deviations. We have indicated this in the Methods-Data analysis section.
Regarding Figure 1, we have removed the means and we are now displaying the medians as blue horizontal lines. Moreover, we decided to remove the black points representing the individual scores as they do not allow us to see the medians in the graphs. In this way, Figure 1 is clearer and cleaner.

8) Conclusions. The abstract improved, well done! However, the conclusions seem not to be conclusions at all, not based on your data.
The first sentence is your (and even mine!) opinion, but it not supported by your data. The second sentence tells about your aims, not your conclusions.
The same is valid for the conclusions at the end of the paper: they show valid comments/assumptions but nothing based on your data.
Useful conclusions should be linked to your aims (to survey editors’ perceptions) and read like “the editors think that”... or “the perceptions of the editors are...”.
We have modified the Conclusions both in the abstract and at the end of the paper, linking them to the survey aims (perceptions, barriers, facilitators…). We have also moved to the end of the Discussion section what was reported in the old Conclusions.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

19 Views

12 Dec 2019 | for Version 2

19 Views Cite this report Responses(0)

Approved

This paper tackles an important issue in science policy research. The authors have responded well to comments on an earlier draft. They have addressed the issues raised and the limitations of the study are clearly stated. While those limitations mean the survey results must be interpreted with caution, the findings are likely to inform discussions around the barriers to and facilitators of interventions to improve adherence to reporting guidelines.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

statistics, modelling, validation, decision making

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

41 Views

26 Nov 2019 | for Version 1

Patricia Logullo, EQUATOR Network, University of Oxford, Oxford, UK

41 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

I cannot comment. A qualified statistician is required.
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Methodology. Reporting quality. Research integrity.

Respond to this report

Responses (1)

Author Response

12 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

We thank you for your time and detailed comments on the manuscript. Please find below a point by point response to them:

Major concerns: methodology

1) Sample characteristics. Although the authors should be praised for the idea of asking editors about interventions to improve reporting, I have a concern about what editors should be surveyed. If one wants to explore the perceived ease of the implementation of one intervention, is it adequate to choose a population of editors whom we know are already “knowledgeable about the topic”?
If these editors are aware of the reporting quality crisis and the reporting guidelines in general, wouldn’t these be more prone to adopt interventions that aim to improve the quality of publications? Therefore, aren’t these the people who would say “yes” to any intervention proposed?
Should the authors not interview editors who have never heard about reporting guidelines too? Wouldn’t this heterogeneous sample be more representative of the scenario of difficulties — or the ease — of implementing reporting guidelines checks in the editorial process? Are the study results only applicable to journals that already endorse or require reporting guidelines?
I would like to see comments about this limitation in the discussion if the authors agree with this or even if they do not.

Firstly, we have now included some more information in the “Participants” section to make it clear what kind of participants we were aiming to reach. Secondly, we have discussed in the Discussion section the other points you mention.

2) Sample size. My second most important concern is about sample size. In the Methods section, the authors do not explain how they planned the sample size and based on what assumptions. 24% seems to be a very low response rate. Do the authors believe 24 is enough? This limitation should be discussed.
We have now discussed these points in the Discussion section.

3) Effectiveness. When you asked your participants about their opinions on “how effective” something would be, what was the definition of effectiveness? Would it be their perception about the intervention improving the adherence to reporting guidelines checklists? Was it a more subjective view of “manuscript quality”?
Thanks to your comment, we have just realised that we only mentioned that we were talking about “effectiveness at improving adherence to RGs” in the methods section, but not in the abstract and introduction. We have now clarified that in the abstract and introduction, and also in some parts of the methods, results and discussion to remind the readers that we refer to effectiveness at improving adherence to RGs.

4) Statreviewer. The authors mention in the text, in Methods:
“Based on feedback from the pilot we decided not to include the intervention “Implementation of the automatic tool Statreviewer14” since participants were not aware of this software and stated that their perceptions would strongly depend on details about how it operates which are not publicly available.”
However, on page 7, Results, they state:
“The implementation of reading tools that automatically assess adherence to RGs, such as Statreviewer14, were seen as potentially interesting interventions.”
After all, was this intervention Statreviewer evaluated (as in Results) or was it not (as in Methods)? Do you mean they mentioned other tools, such as Penelope, GoodReports, Cobweb or others? Or were they referring to Statreviewer?
We have clarified in the results section (“Further interventions and incentives for authors and journals”) that Statreviewer was not included in the survey but mentioned by some participants when asked about further potentially effective interventions.

Major concerns: reporting
5) Journals. You interviewed 24 editors from 20 journals. What are these journals? I did not see a list of their names, and I believe it is important.
The 20 journals represented are now listed in Table 1. We have also improved the description of the sample in the first paragraph of the Results section.

6) Figure 1. Figure 1 is difficult to read/interpret. I like the idea of having the two outcomes (ease of implementation and effectiveness) in one figure very much so that one can compare the two. However, the bar chart is not clear at all, nor easy to interpret. Colours do not help. The authors should consider building a figure with plot points (which would be easy, given the small sample) showing where each participant sits within the Likert scale — not bars.
We have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.

Another suggestion for Figure 1 (here, just a suggestion) is to add something to remind the readers what the interventions are. As Box 1 is far from the Figure, it is difficult to remember what each intervention is. It would be useful to have “nicknames”, for example:
1. Page numbers
2. Checklist
3. Highlight
4. Subheadings
5. COBWEB
6. Use RG
7. Check RG
8. Staff
9. Training
We have incorporated similar nicknames to help readers remember the interventions.

7) Peer reviewers’ checks. The abstract is, in general, well written and clear. However, I got intrigued by this sentence:
“Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and could decrease the overall quality of reviews.”
I was surprised that peer reviewers checking reporting guidelines adherence would decrease the overall quality of the paper. Then I reread it in the paper, but here put in a complete sentence, that makes sense:
“Participants were concerned that the quality of peer review could be compromised as reviewers should focus on the manuscript’s content and not on the reporting issues.”
Meaning, I suppose, that peer reviewers would focus on checking reporting guidelines and not on their “expert view” of the study. I get it, but it is not very clear, neither in the text, nor in the abstract, and I invite the authors to rewrite both. Especially because “reporting guidelines” are about content too. They point out the content elements that should not be absent from papers, so using the word “content” in the manuscript text is not really appropriate. Please, re-evaluate.
We completely agree with the point you raise. We have re-written this point in the Abstract, Results and Conclusions. “Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and were concerned that the quality of peer review could be compromised. Reviewers are generally not expected to focus on reporting issues but on providing an expert view on the importance, novelty, and relevance of the manuscript.”

8) As I said above, the abstract is good, and there is room for growth (you have now 232 words). However, I could not find the conclusions in the abstract helpful. Could you please inform better what you did conclude? What “points raised in the survey” should be considered?
We have re-written and expanded the Conclusions in the abstract. The abstract is now 295 words.

9) Checklists as forms. ...“assessing completeness of reporting is a subjective task”. This is one of the most important sentences in this manuscript, it tells us about using reporting guidelines checklists as evaluation tools — for what they have not been created. I saw no discussion about this in your paper.
We have now discussed this point in the paragraph before the limitations of the study.

10) Discussion. The discussion of the paper is short, and it lacks several points pointed above. Mainly, it does not discuss the limitations and potential bias of this study. Discussing limitations is a requirement from F1000Research journal.
We have now included such a section in the Discussion section.

11) You mention you did analyse qualitative information using NVivo. How did you do it?
We have now provided further information on how we did the qualitative analysis with NVivo (“Data analysis” section).

12) There are at least eight items from the CHERRIES checklist that I could not see in your paper, and you might want to report, even if in a separate (or supplementary) box, from data protection, preventing multiple entries, to randomisation of items, and others.
We have now included the information corresponding to the missing items throughout the whole Methods section.

13) From Box 1, I do not quite get the difference between Interventions 1 and 2. It seems that both ask authors to submit a populated/completed RG checklist. Is it that #1 asks for page numbers and #2 does not?
#1 asks for page numbers and #2 for text from the manuscript. Some people may think that if you nudge authors to have to copy-paste text into the checklist they may tend to take it more seriously than if you just ask them to put a page number. We have slightly modified their wording in Box 1 to make it clearer.

14) In my opinion, the sentences from “To contact editors...” to “improve adherence to RGs” belong more to the “Participants” section than to the “Procedure”. Especially the first sentence. Also “Participants could suggest further editors that they considered could contribute to the survey” is something that I would consider as a snowballing or recruitment method, not the survey procedure per se. So, I would suggest putting everything related to recruitment together in one section, to make it easier for your reader (who might be interested in undertaking a similar survey) to find information on such a critical issue as recruitment. Also: how could participants suggest? Was it a specific question in the SurveyMonkey form?
We have restructured that part of the manuscript and divided it into “Recruitment” and “Survey administration”. We have also provided the information requested.

15) Were participants given an estimation of the time required to answer the survey before they began?
We have now included this information (“Survey administration” subsection).

Minor: about citations
16) Reference number 1 (EQUATOR Network website) is not a proper, complete reference. It points to the home page of the website, not to a document, and as so, it does not show from where the authors collected that information (the reader cannot find them in the home page). The correct way of citing would be adding the full URL. A suggestion: EQUATOR Network. What is a reporting guideline? Available at:
http://www.equator-network.org/about-us/what-is-a-reporting-guideline/. Cited on 19/11/2019.
Done.

17) Something similar happens to reference number 13. Author and title are missing. It would help readers if it could be typed like this:
Schoter S. Evidence-based publishing. BMJ. Available at: https://www.bmj.com/about-bmj/evidence-based-publishing. Cited on 19/11/2019.
Done.

18) Reference 14 also lacks authors and titles. References 13, 14, 16, 21 and 22 lack a lot of elements necessary for readers to be able to find them. When working in a printed version (as I am now), you cannot see what the document is, where it is published or by whom — these are available only by those reading online and able to “click” on the links.
Done.

19) I would change the sentence:
“Raw survey results are given as Underlying data20.”
To something clearer and that does not require the reader to go to the reference list to understand what it is.
“Table XX in the supplementary file X20 shows the responses from all 24 participants.”
Done.

20) Has the study by Hawwash et al. (reference 25) been published yet? (The authors cited the protocol only). Do the authors plan to cite other electronic tools that are out there?
We have now cited the results paper by Hawwash et al., which got published on 6 November. Regarding the electronic tools, we have already cited COBWEB and Statreviewer. We did not cite Penelope as it does not explicitly check RG requirements – it just helps authors submit the relevant RG checklist.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

31 Views

20 Nov 2019 | for Version 1

Gerard Urrutia, Iberoamerican Cochrane Center, Sant Pau Biomedical Research Institute, Barcelona, Spain

31 Views Cite this report Responses(1)

Approved

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Systematic reviews

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

12 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

However, the study includes a limited sample of participants (24 editors belonging to 20 medical journals), which limits the scope of the conclusions. Although the study, according to its stated objectives, does not intend to offer a representative view of the editors of all medical journals, the limited number of participants as well as their representativeness (the sources from which they were identified and selected allows us to think that it is a highly selected sample of publishers with previous experience and a potential motivation in the subject), is not mentioned in the discussion.

We have now discussed these points in the Discussion section.

It is noteworthy that, despite this potentially "favourable" sample, the response rate is very low (24%). What would have been the participation rate and the responses if a larger and more representative sample had been selected? Although this was not an objective of the study, it would deserve some reflection in the discussion.

We have now included some comments about the response rate in a new paragraph on Limitations (Discussion section).

It would also be useful to provide details on the origin of the participants in the survey according to the sources of selection, which would improve the description of the sample.

We have now added that information to the first paragraph of the Results section.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

37 Views

11 Nov 2019 | for Version 1

37 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

statistics, modelling, validation, decision making

Respond to this report

Responses (1)

Author Response

12 Dec 2019

David Blanco, Université de Paris, CRESS, INSERM, INRA, Paris, France

We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

A limitation of the study is that the results are based upon a survey return rate of just 24%. The returns are likely biased favouring editors with stronger views on reporting guidelines.

We have now reflected on this point in the Discussion section.

In addition, it is not clear whether all ten of the top-ten journals were represented, or whether one or more journals dominate the response rate. In addition it is not completely clear how many participants came from the MiRoR Network, or journals previously publishing studies on Reporting Guidelines.

We have now added that information to the first paragraph of the Results section.

Figure 1 is hard work to interpret and might be presented more usefully as two separate graphs. In addition, the colour scheme could be changed to highlight those interventions considered "Very easy" or "Moderately easy" to implement, and those ranked as "Very effective" or "Moderately effective". (Note that the proportions of each score could still be retained.)

In order to make Figure 1 easier to interpret, we have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.
However, we prefer not to separate it into two graphs as we believe it is interesting for the readers to be able to compare the scores that each intervention obtains for each of the two outcomes.

The paper would be improved by a paragraph on Limitations of the Study.

We have now included a section on Limitations of the study in the Discussion section.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. EQUATOR Network: EQUATOR Network website-the resource centre for. 2013. Reference Source

[2] 2. Begg C, Cho M, Eastwood S, et al.: Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996; 276(8): 637–9. PubMed Abstract | Publisher Full Text

[3] 3. Samaan Z, Mbuagbaw L, Kosa D, et al.: A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013; 6: 169–188. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Blanco D, Altman D, Moher D, et al.: Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open. 2019; 9(5): e026589. PubMed Abstract | Publisher Full Text | Free Full Text

[5] 5. Shamseer L, Stevens A, Skidmore B, et al.: Does journal endorsement of reporting guidelines influence the completeness of reporting of health research? A systematic review protocol. Syst Rev. 2012; 1(1): 24. PubMed Abstract | Publisher Full Text | Free Full Text

[6] 6. Stevens A, Shamseer L, Weinstein E, et al.: Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review. BMJ. 2014; 348: g3804. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Blanco D, Biggane AM, Cobo E, et al.: Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers? Trials. 2018; 19(1): 80. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Hair K, Macleod MR, Sena ES, et al.: A randomised controlled trial of an Intervention to Improve Compliance with the ARRIVE guidelines (IICARus). Res Integr Peer Rev. 2019; 4(1): 12. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Schulz KF, Altman DG, Moher D, et al.: CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010; 340: c332. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Moher D, Liberati A, Tetzlaff J, et al.: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009; 339: b2535. PubMed Abstract | Publisher Full Text | Free Full Text

[11] 11. von Elm E, Altman DG, Egger M, et al.: Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007; 335(7624): 806–8. PubMed Abstract | Publisher Full Text | Free Full Text

[12] 12. Chan AW, Tetzlaff JM, Gøtzsche PC, et al.: SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013; 346: e7586. PubMed Abstract | Publisher Full Text | Free Full Text

[13] 13. Evidence based publishing. Reference Source

[14] 14. Statreviewer. Reference Source

[15] 15. Blanco D: Extended data of the project "A survey exploring biomedical editors' perceptions of editorial interventions to improve adherence to reporting guidelines" [Data set]. Zenodo. 2019. http://www.doi.org/10.5281/zenodo.3404002

[16] 16. R: A language and environment for statistical computing. 2019. Reference Source

[17] 17. NVivo qualitative data analysis software. 2018. Reference Source

[18] 18. Eysenbach G: Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004; 6(3): e34. PubMed Abstract | Publisher Full Text | Free Full Text

[19] 19. Tong A, Sainsbury P, Craig J: Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Heal Care. 2007; 19(6): 349–57. PubMed Abstract | Publisher Full Text

[20] 20. Blanco D: Underlying data of the project "A survey exploring biomedical editors' perceptions of editorial interventions to improve adherence to reporting guidelines" [Data set]. Zenodo. 2019. http://www.doi.org/10.5281/zenodo.3407725

[21] 21. Health NI of: What is CME Credit? Reference Source

[22] 22. EQUATOR Network Toolkits. Reference Source

[23] 23. Cobo E, González JA: Taking advantage of unexpected WebCONSORT results. BMC Med. 2016; 14(1): 204. PubMed Abstract | Publisher Full Text | Free Full Text

[24] 24. Pandis N, Shamseer L, Kokich VG, et al.: Active implementation strategy of CONSORT adherence by a dental specialty journal improved randomized clinical trial reporting. J Clin Epidemiol. 2014; 67(9): 1044–8. PubMed Abstract | Publisher Full Text

[25] 25. Hawwash D, Lachat C, De Cock N, et al.: Integrating a writing aid to facilitate the use of reporting guidelines: A crossover randomized controlled trial Trial protocol. Reference Source

[26] 26. Publons: Global State of Peer Review 2018. Reference Source

[27] 27. Vilaró M, Cortés J, Selva-O’Callaghan A, et al.: Adherence to reporting guidelines increases the number of citations: the argument for including a methodologist in the editorial process and peer-review. BMC Med Res Methodol. 2019; 19(1): 112. PubMed Abstract | Publisher Full Text | Free Full Text

[28] 28. Blanco D, Schroter S, Moher D, et al.: Evaluating the Impact of Assessing During Peer Review the CONSORT Checklist Submitted by Authors. Reference Source

A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines

Abstract

Keywords

Abbreviations

Introduction

Methods

Participants

Procedure

Questionnaire development

Box 1. Interventions included and their targets.

Data analysis

Ethics approval & informed consent

Reporting guidelines

Results

Table 1. Demographic characteristics of the 24 participants.

Current practice

Perceptions of nine potential interventions

Figure 1. Scores for perceived ease of implementation and perceived effectiveness.

Identifying the barriers and facilitators

Further interventions and incentives for authors and journals

Discussion

Data availability

Underlying data

Extended data

Acknowledgments

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated