F1000Research F1000Research 2046-1402 F1000 Research Limited London, UK 10.12688/f1000research.133688.1 Study Protocol Articles Current trends, barriers, and facilitators of use of core outcome sets in Cochrane systematic reviews: Protocol

[version 1; peer review: 2 approved with reservations]

 Saldanha Ian Conceptualization Data Curation Formal Analysis Funding Acquisition Investigation Methodology Project Administration Resources Supervision Validation Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0002-2062-3382 a 1 Hughes Karen Data Curation Formal Analysis Methodology Software Writing – Review & Editing 2 Dodd Susanna Data Curation Formal Analysis Methodology Writing – Review & Editing 2 Lasserson Toby Validation Writing – Review & Editing 3 Kirkham Jamie Formal Analysis Methodology Validation Writing – Review & Editing 4 Lucas Samuel Data Curation Investigation Writing – Review & Editing 2 Williamson Paula Conceptualization Data Curation Formal Analysis Investigation Methodology Project Administration Resources Software Supervision Validation Writing – Review & Editing 2 Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA Department of Health Data Science, University of Liverpool, Liverpool, England, UK Central Editorial Unit, Cochrane, London, England, UK Centre for Biostatistics, The University of Manchester, Manchester, England, UK isaldan1@jhu.edu

No competing interests were disclosed.

 23 6 2023 2023 12 735 28 4 2023 Copyright: © 2023 Saldanha I et al. 2023 This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Background: Core outcome sets (COS) represent agreed-upon minimum outcomes that should be reported in all studies in a given topic area. Cochrane reviews are considered among the most rigorously conducted systematic reviews (SRs). In 2019, seven of the first 100 published Cochrane SRs (7%) cited a COS in relation to choosing outcomes. A relevant COS existed but was not mentioned (or cited) for 27 of the remaining 93 SRs (29%). Among Cochrane Review Group editors surveyed in 2019, 86% felt that COS should definitely/possibly be used in Cochrane SRs. As of September 2019, the Cochrane Handbook recommends that SR teams consult resources that host relevant COS when choosing outcomes for the SR.

Objectives: (1) Examine the extent to which authors are currently considering COS to inform outcome choice in Cochrane protocols and completed SRs. (2) Understand author barriers and facilitators of using COS in Cochrane protocols and completed SRs.

Methods: We will examine the extent to which all Cochrane SRs published in the last 3 months of 2022 and all Cochrane protocols published in 2022: (a) cited a COS, (b) searched for COS, and (c) reported outcome inconsistency among included studies and/or noted the need for COS. One investigator will extract information from SRs and protocols; a second extractor will verify all information, discussing discrepancies to achieve consensus. Using Jisc Online Surveys ®, we will conduct an online anonymous survey of authors of all the included completed SRs and protocols to assess author awareness of COS and identify barriers and facilitators of using COS to inform outcome choice.

Discussion: This study will provide key information regarding uptake of COS by Cochrane SR authors and the barriers and facilitators that they experience. Our findings will inform approaches to increasing awareness and uptake of COS in future SRs, both with and beyond Cochrane.

 core outcome sets systematic reviews Cochrane outcome choice barriers facilitators The author(s) declared that no grants were involved in supporting this work. Background

Core outcome sets (COS) represent agreed-upon minimum outcomes that should be reported in all research studies in a given topic area. 1 COS are intended to increase the usefulness of research evidence and facilitate comparisons across studies. Although COS have traditionally been developed for use in clinical trials, increasingly COS are also developed for all research as well as other parts of the evidence ecosystem, such as for routine care. 2 , 3 A systematic review (SR) is a research effort whose goal is to identify and synthesize all relevant studies that fulfill pre-specified eligibility criteria to answer a specific research question(s). 4 , 5 Thus, systematic reviewers not only evaluate primary research that should have considered COS 6 but they should themselves also consider COS when choosing outcomes for the SR.

Cochrane SRs are considered among the most rigorously conducted SRs in the world. We previously conducted an analysis and reported that only seven of the first 100 published Cochrane SRs in 2019 (7%) cited a COS in relation to choosing outcomes. 7 A relevant COS existed but was not mentioned (or cited) for 27 of the remaining 93 SRs (29%). For a further 6 reviews, a relevant COS was published after the protocol of the SR was developed. 7 We similarly conducted an analysis of all 67 comparative effectiveness SRs that were published between 2018 and 2020 by Evidence-based Practice Centers (EPCs) with funding from the U.S. Agency for Healthcare Research and Quality (AHRQ) (EPC SRs are also highly rigorous SRs). 8 We found relevant COS for 36 of the 67 EPC SRs (54%). 8 In the Cochrane analysis paper, we also reported results of a survey of editors of 36 (of the then 52) Cochrane Review Groups; most editors (31/36; 86%) felt that COS should definitely/possibly be used in Cochrane SRs. 7

Considerations regarding COS scope and contextual relevance to the SR notwithstanding, there is no obvious reason why COS should not be used when choosing outcomes to be examined in SRs. 8 Use of existing COS represents an opportunity for systematic reviewers to leverage the efforts of COS developers in identifying outcomes for their SRs. Indeed, the 2 nd (i.e., most recent) edition of the Cochrane Handbook for Systematic Reviews of Interventions now recommends that SR teams consult resources that host relevant COS when choosing outcomes for the review. 9 However, this edition of the Handbook was first published in September 2019, which was after outcome choices were made for all SRs in the sample of Cochrane SRs published in 2019 that we analyzed. Therefore, the extent to which recent Cochrane SRs (and protocols) are considering COS during outcome choice is unclear.

Use of COS to inform choice of outcomes can be considered a behavior. 10 In 2022, based on a survey of authors of clinical trials submitted to the top five medical journals (in terms of impact factor), Matvienko-Sikar and colleagues reported that the most common barrier to COS use was trialist preferences and choices regarding outcomes (68% of respondents), and the most common facilitator was trialist awareness and knowledge about COS (90%). 11 Also in 2022, Hughes and colleagues conducted qualitative interviews of clinical trialists in the UK and reported that the biggest barriers to COS uptake were trialist perceptions regarding COS characteristics (e.g., increased patient burden, COS being out of date) and the COS development process (e.g., the lack of inclusion of all relevant clinical specialties). 12 The biggest facilitators were trialist awareness and understanding of COS and funder and journal editor recommendations to use COS. 12

Outcomes chosen for trials and for SRs, even within the same topic area, have been shown sometimes to be inconsistent, perhaps reflecting differing priorities. 13 , 14 Moreover, for Cochrane SRs, these decisions are often shared between the authors and the editorial team (with input from the peer reviewers, including people with lived experience of the condition of interest). 7 It may be that these various parties have different priorities, although the SR authors are primarily responsible for choosing which outcomes to assess and this will likely influence the final set of outcomes for a Cochrane SR. The stages of protocol development and subsequent peer review help vet the choice of outcomes.

We are aware of two surveys assessing barriers to COS use by SR authors, specifically in the topic area of pain. 15 , 16 Boric and colleagues surveyed authors of SRs of interventions for neuropathic pain and reported that the main barrier to use of the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) COS was the lack of awareness of the full COS. 15 Similarly, Dosenovic and colleagues surveyed authors of SRs of interventions for postoperative pain in children and reported that the main barriers to use of the pediatric version of the IMMPACT COS (PedIMMPACT) were the lack of awareness, difficulties with implementation, and the lack of resources. 16

To our knowledge, the barriers and facilitators of use of COS have not been examined among Cochrane SR authors (in our previous analyses, we did not survey the authors of either the Cochrane SRs or EPC SRs). The insights that could be gained from such an examination would be crucial to inform approaches to increasing awareness and uptake of COS in future SRs, both with and beyond Cochrane.

 Objectives

This study has two objectives:

Examine the extent to which authors are currently considering COS to inform outcome choice in Cochrane protocols and completed SRs.

Understand author barriers and facilitators of using COS in Cochrane protocols and completed SRs.

Methods Summary of methods

For Objective 1, we will assess recent Cochrane SRs for the extent to which they (a) cited a COS, (b) mentioned searching for COS, and (c) reported outcome inconsistency among included studies and/or noted the need for COS. For Objective 2, we will conduct an online anonymous survey of authors of these recent Cochrane SRs to identify barriers and facilitators of their using COS to inform outcome choice in SRs.

 Methods for Objective 1

Eligibility Criteria for Cochrane SRs: We will include all completed Cochrane SRs published in the last 3 months of 2022 and all Cochrane SR protocols published throughout 2022. This period restriction will enable us to examine contemporary practices regarding outcome choice in Cochrane SRs. We will not restrict SRs or protocols by topic area or location of the authors or Cochrane Review Group.

Identifying Cochrane SRs: We will identify relevant completed SRs and protocols by searching the Cochrane Database of Systematic Reviews through the Cochrane Library, restricting by the periods of interest.

Data extraction:

For all relevant SRs, we will extract information about whether the SR authors:

Cited a COS in the context of choosing outcomes for the review, and

Mentioned having searched for a COS to choose outcomes for the review.

For completed SRs only, we will extract information about whether the SR authors:

Noted any problems with outcome inconsistency across the included studies and/or the need for outcome standardization/COS development.

For all SRs that cited or mentioned using a COS (i.e., yes to #1 or #2 above), we will also extract:

The extent to which the SR authors used outcomes from the COS. In other words, we will assess the extent of overlap between the COS outcomes and the SR outcomes using a framework that we developed 8 and has been used since. 17 Briefly, we will focus on the outcome domains (the “what,” e.g., pain) but will not examine whether the “how” of the outcome (e.g., one instrument for measuring pain versus another) matched. The framework we developed considers matches to be general (i.e., nonspecific) or specific. Our approach to determining the type of match for pairs of outcomes is consistent with an approach that has been used previously. 3 , 17 , 18

For SRs that did not cite or mention having considered a COS (i.e., no to both #1 and #2 above), we will also extract whether:

A relevant COS exists. We will do this by searching the Core Outcome Measures for Effectiveness Trials (COMET) database. Maintained by the COMET Initiative, this is a free, online, regularly updated, searchable database of COS. One investigator will assess the potential relevance of each identified COS to the topic of each SR following an approach used previously. 7 , 17 , 19 For identified COS, we will also extract information regarding when the COS was published. This will allow us to assess whether the COS was available during the stage of outcome choice for the SR/protocol.

For information extraction, we will review all sections of the SR/protocol report. One investigator will extract information from each SR/protocol and a second investigator will independently verify the extracted information.

Statistical analysis: We will calculate descriptive statistics (percentages and medians with interquartile ranges [IQRs]). We will conduct all data analyses using Stata Version 16 (College Station, Texas, USA).

 Methods for Objective 2

Eligibility Criteria for Cochrane SR authors: We will email a survey to the corresponding authors of all completed relevant Cochrane protocols and completed SRs from Objective 1. If a corresponding author does not respond to the survey or our email, we will send the survey to the senior (i.e., last) author. If similar nonresponse occurs, we will send the survey to another author from the author list (e.g., first author). At each request, we will suggest that the person(s) most familiar with the considerations during outcome choice for the SR complete a single response for the SR.

Survey design and implementation: We will design and distribute the anonymous survey using Jisc Online Surveys ® in English. We will send the author(s) of each relevant SR one of four versions of the survey:

Version A: For SRs that cited/mentioned searching for a COS and the full COS was used.

Version B: For SRs that cited/mentioned searching for a COS and some but not all COS outcomes were used.

Version C: For SRs that did not cite/mention searching for a COS, but we identified a potentially relevant COS that could have been used.

Version D: For SRs that did not cite/mention searching for a COS and we did not identify a potentially relevant COS that could have been used.

Items of interest for the surveys include:

Author process for outcome choice for the SR

Awareness of COS in general

Whether the authors searched for COS (regardless of whether the SR mentions that the authors searched for it)

Awareness of a relevant COS (if one exists for the SR)

Reasons for non-use of relevant COS (if one exists for the SR)

In instances where a COS exists but was not used, we will also ask authors whether they will consider incorporating the unused outcomes from the COS when completing the SR (for protocols) or when updating the review (for completed SRs).

Perceived barriers to COS use

Perceived facilitators of COS use.

For items 5, 6, and 7 above, Box 1 lists examples of prespecified as well as open-ended response options.

Prespecified and open-ended response options for three specific items in the survey for Objective 2: reasons for non-use of existing COS, general barriers to COS use, and general facilitators of use of COS in SRs.

Reasons for non-use of existing COS

Did not know there was a COS.

COS target population was too narrowly/broadly defined for the SR population of interest.

COS target intervention was too narrowly/broadly defined for the SR intervention of interest.

COS was too old.

Other, specify: _____________

General barriers to use of COS in SRs.

Systematic reviewers’ preference to use their own outcomes

Inadequate knowledge about the existence of COS

Inadequate knowledge about how to use core outcomes

Poor quality and design of COS

Systematic reviewers were not involved in COS development

Belief that primary study outcomes are different from SR outcomes

Other, specify: _____________

General facilitators of use of COS in SRs.

Use of COS can facilitate incorporating more studies into meta-analyses

Positive perceptions of COS

Good knowledge about the existence of COS

Good knowledge about how to use core outcomes

Availability of well-designed COS

Belief that primary study outcomes should be fundamentally akin to SR outcomes

SR funder recommended COS use

Guideline developer (that the SR was meant to inform) recommended COS use

Other, specify: _____________

Institutional Review Board (IRB) approval: Before we distribute the surveys, we will obtain Research Ethics Committee approval from the University of Liverpool, UK.

Data analysis: We will analyze the data by calculating descriptive statistics. For open-ended responses to the survey, we will use content analysis. For overlapping questions across versions of the survey, we will analyze and report overall results as well as separate results by survey version. We will report information from the survey only in the aggregate.

 Dissemination of findings

We will disseminate the findings regarding both objectives of this study through publication of peer-reviewed manuscripts and presentation at international conferences, such as the Cochrane Colloquium.

 Study status

We have identified the Cochrane SRs and protocols and are currently extracting information for the study.

 Discussion

Use of COS in Cochrane SRs is important to improve outcome standardization, reduce research waste, and improve evidence synthesis regarding the effects of interventions in particular health areas. This study will provide useful findings regarding the extent of uptake of COS by Cochrane reviewers and key insights regarding author views and reasons for using, or not using, COS when deciding on outcomes for Cochrane SRs. The findings and insights are essential to better understand barriers and facilitators of COS uptake in Cochrane SRs, so that interventions to promote uptake can be developed and improved.

 Data availability

No data are associated with this article.

 References Williamson PR Altman DG Blazeby JM : Developing core outcome sets for clinical trials: issues to consider. Trials. Aug 6 2012;13:132. 22867278 10.1186/1745-6215-13-132 PMC3472231 Kearney A Gargon E Mitchell JW : A systematic review of studies reporting the development of Core Outcome Sets for use in routine care. J. Clin. Epidemiol. Mar 20 2023;158:34–43. 36948407 10.1016/j.jclinepi.2023.03.011 Dodd S Harman N Taske N : Core outcome sets through the healthcare ecosystem: the case of type 2 diabetes mellitus. Trials. Jun 25 2020;21(1):570. 32586349 10.1186/s13063-020-04403-1 PMC7318375 Institute of Medicine. Finding What Works in Health Care: Standards for Systematic Reviews. The National Academies Press;2011;340. Lasserson TJ Thomas J Higgins JPT : Chapter 1: Starting a review. Higgins JPT Thomas J Chandler J , editors. Cochrane Handbook for Systematic Reviews of Interventions version 60. 2nd ed. John Wiley & Sons;2019. Williamson PR Barrington H Blazeby JM : Review finds core outcome set uptake in new studies and systematic reviews needs improvement. J. Clin. Epidemiol. Oct 2022;150:154–164. 10.1016/j.jclinepi.2022.06.016 Williamson PR Ávila OR de Clarke M : Assessing the relevance and uptake of core outcome sets (an agreed minimum collection of outcomes to measure in research studies) in Cochrane systematic reviews: a review. BMJ Open. Sep 6 2020;10(9):e036562. 32895272 10.1136/bmjopen-2019-036562 PMC7476465 Saldanha IJ Dodd S Gorst SL : More than half of systematic reviews have relevant core outcome sets. J. Clin. Epidemiol. Aug 2021;136:168–179. 33974970 10.1016/j.jclinepi.2021.04.019 PMC8442852 Higgins JPT Thomas J Chandler J : Cochrane Handbook for Systematic Reviews of Interventions version 6.1. 2nd ed. John Wiley & Sons;2019. Matvienko-Sikar K Byrne M Clarke M : Using behavioural science to enhance use of core outcome sets in clinical trials: protocol. [version 1; peer review: 1 approved]. HRB Open Res. 2022;5(23). 10.12688/hrbopenres.13510.1 Matvienko-Sikar K Avery K Blazeby JM : Use of core outcome sets was low in clinical trials published in major medical journals. J. Clin. Epidemiol. Feb 2022;142:19–28. 34715310 10.1016/j.jclinepi.2021.10.012 Hughes KL Williamson PR Young B : In-depth qualitative interviews identified barriers and facilitators that influenced chief investigators' use of core outcome sets in randomised controlled trials. J. Clin. Epidemiol. Apr 2022;144:111–120. 34896233 10.1016/j.jclinepi.2021.12.004 PMC9094758 Saldanha IJ Li T Yang C : Clinical trials and systematic reviews addressing similar interventions for the same condition do not consider similar outcomes to be important: a case study in HIV/AIDS. J. Clin. Epidemiol. Apr 2017;84:85–94. 28249722 10.1016/j.jclinepi.2017.02.005 PMC5441957 Saldanha IJ Lindsley K Do DV : Comparison of Clinical Trial and Systematic Review Outcomes for the 4 Most Prevalent Eye Diseases. JAMA Ophthalmol. Sep 1 2017;135(9):933–940. 28772305 10.1001/jamaophthalmol.2017.2583 PMC5625342 Boric K Boric M Dosenovic S : Authors' lack of awareness and use of core outcome set on postoperative pain in children is hindering comparative effectiveness research. J. Comp. Eff. Res. May 2018;7(5):463–470. 29775075 10.2217/cer-2017-0079 Dosenovic S Nikolic Z Ivancev B : Awareness and acceptability of Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials core outcome set for chronic pain among surveyed neuropathic pain authors. J. Comp. Eff. Res. Jul 2019;8(9):671–683. 31290689 10.2217/cer-2018-0123 Saldanha IJ Dodd S Fish R : Representation of core outcomes in regulatory guidance from the FDA and EMA. BMJ Medicine. 2022;1:e000233. 36936602 10.1136/bmjmed-2022-000233 PMC9978677 Dodd S Fish R Gorst S : Representation of published core outcome sets for research in regulatory guidance: protocol. HRB Open Res. 2021;4:45. 10.12688/hrbopenres.13139.3 Hughes KL Kirkham JJ Clarke M : Assessing the impact of a research funder's recommendation to consider core outcome sets. PLoS One. 2019;14(9):e0222418. 31518375 10.1371/journal.pone.0222418 PMC6743767 10.5256/f1000research.146696.r191471 Reviewer response for version 1 Chou Roger 1 Referee Departments of Medicine, and Medical Informatics & Clinical Epidemiology, Oregon Health and Science University Foundation, Portland, Oregon, USA

Competing interests: No competing interests were disclosed.

 31 8 2023 Copyright: © 2023 Chou R 2023 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

This is a protocol for a study evaluating the extent to which Cochrane reviews utilize core outcomes sets, and to understand barriers and facilitators to use of core outcome sets through author surveys. The objectives and methods are clear and the study will provide interesting and useful information to inform future systematic reviews and guidance on reviews.

Re: the methods, for reviews that don't use a core outcome set, the study basically focuses on whether a core outcome set is available. In cases where there is a core outcome set, I think it would be useful and perhaps more meaningful to also compare the degree to which the review evaluated outcomes in the core outcome set i.e. even if they don't cite a core outcome set they could have evaluated most or all of the outcomes.

I also think it is important to consider that for some Cochrane review groups (e.g., Back and Neck) there is a set of outcomes that authors are asked to include. This is not a formal "core outcome set" but may be utilized like one (and these outcomes are often based on core outcome sets). In these cases I am not sure that not citing/using a core outcome set is necessarily problematic.

In terms of barriers to use of core outcome sets, one thing that comes up is that Cochrane/GRADE restrict the number of outcomes to be evaluated. This can create issues when a core outcomes specifies many outcomes.

Regarding reasons for not using core outcome sets (Box 1), some reviews could be focused on a particular outcome e.g. a particular harm.

In the Methods would be useful to describe the number of reviews/authors that are expected to be included, to give some sense of how robust the data and surveys will be.

Is the study design appropriate for the research question?

Partly

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

Systematic reviews and systematic review methods

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Saldanha Ian Epidemiology, Johns Hopkins University, Baltimore, Maryland, USA

Competing interests: No competing interests were disclosed.

 20 9 2023

This is a protocol for a study evaluating the extent to which Cochrane reviews utilize core outcomes sets, and to understand barriers and facilitators to use of core outcome sets through author surveys. The objectives and methods are clear and the study will provide interesting and useful information to inform future systematic reviews and guidance on reviews.

Response: We thank Dr. Chou for reviewing this manuscript and providing useful comments.

Re: the methods, for reviews that don't use a core outcome set, the study basically focuses on whether a core outcome set is available. In cases where there is a core outcome set, I think it would be useful and perhaps more meaningful to also compare the degree to which the review evaluated outcomes in the core outcome set i.e. even if they don't cite a core outcome set they could have evaluated most or all of the outcomes.

Response: We believe in the second sentence of this comment, Dr. Chou is referring to cases where there isn’t a core outcome set cited. We agree with the suggestion and have modified the text in the Data Extraction section of the Methods accordingly.

I also think it is important to consider that for some Cochrane review groups (e.g., Back and Neck) there is a set of outcomes that authors are asked to include. This is not a formal "core outcome set" but may be utilized like one (and these outcomes are often based on core outcome sets). In these cases I am not sure that not citing/using a core outcome set is necessarily problematic.

Response: We agree that this is likely the case for some Cochrane review groups. We anticipate that this sentiment will come through during the surveys when we ask authors what influenced the choice of outcomes for the SRs. Regardless, we believe that whether core outcome sets are used or not used should be clearly reported. Searching for core outcome sets is in keeping with current guidance in the Cochrane Handbook for Systematic Reviews of Interventions.

In terms of barriers to use of core outcome sets, one thing that comes up is that Cochrane/GRADE restrict the number of outcomes to be evaluated. This can create issues when a core outcomes specifies many outcomes.

Response: We agree that this may be the case for some core outcome sets but not most (the average number of outcomes in core outcome sets is about 6). It will be of interest to see if this is stated as an ‘other’ barrier by survey respondents.

Regarding reasons for not using core outcome sets (Box 1), some reviews could be focused on a particular outcome e.g. a particular harm.

Response: We don’t agree that being made to include a particular harm precludes a systematic review author from considering core outcome sets. By definition, a core outcome set is a minimum set of outcomes. Systematic reviewers should feel free to include additional outcomes. But, to Dr. Chou’s earlier comment, there may be restrictions to the number of outcomes in the systematic review.

In the Methods would be useful to describe the number of reviews/authors that are expected to be included, to give some sense of how robust the data and surveys will be.

Response: We have inserted the following sentence at the end of the Eligibility Criteria for Cochrane SRs section of the Methods: “A total of 294 SRs are eligible for Objective 1.”

 10.5256/f1000research.146696.r194904 Reviewer response for version 1 Sipe Theresa 1 Referee https://orcid.org/0000-0002-4725-0432 Retired from Centers for Disease Control and Prevention, Atlanta, USA

Competing interests: No competing interests were disclosed.

 29 8 2023 Copyright: © 2023 Sipe T 2023 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

This protocol entitled ‘Current trends, barriers, and facilitators of use of core outcome sets in Cochrane systematic reviews: Protocol’  is describing a protocol of a study that will examine systematic review (SR) protocols and completed reviews for inclusion of core outcome sets (COS) as well as conduct a survey of SR protocol/review authors for barriers and facilitators of using COS.

In general, the protocol is well written and thorough in descriptions. There is some repetition and long sentences that could be reduced. The background does a good job explaining COS and some additional context will help readers not familiar with this concept.

Background, p. 3 of 8, 1 st paragraph. Term ‘evidence ecosystem’ needs explanation. Seems that time context is needed. The time context will help with understanding the study done in 2019 that is cited in next paragraph.  

2 nd paragraph – 1 st sentence needs citation.  

3 rd paragraph – 1 st sentence is long and has a double negative, making it confusing. It also is a conclusion statement without evidence. Encouragement to do the action may be a better way to word the sentence or re-word to become common practice. 2 nd sentence - ‘(i.e., most recent)’ may not be necessary. Last sentence is unclear and Cochrane SRs are personified. It would be the extent in which authors of Cochrane SRs/protocols are considering COS.

7 th paragraph, 1 st sentence – the text in parentheses is long and can be shortened ‘(in our previous analyses, we did not survey the authors…)’. Last sentence should ‘both with and beyond Cochrane’ be ‘both within and beyond Cochrane’? If so, this fix is needed for the abstract. 

Methods p. 4 of 8. The design of the study is not stated. It seems to have systematic review components (literature search, inclusion/exclusion criteria, data abstraction) but does not meet all of the requirements for a systematic review. However, it could be considered a rapid review. Regardless, please state what the study design is.

Methods for Objective 1, p. 4 of 8,  eligibility criteria -  Will there be any exclusions?

(4) ‘since’ at the end of the sentence may not be grammatically correct.

Statistical analyses, p. 4 of 8 – the authors stat that medians and IQRs will be used, however this term is often misused. IQRs are a single number (75 th percentile minus 25 th percentile). So maybe the authors mean ‘interquartile interval’ or more simply 25 th and 75 th percentiles.

Eligibility Criteria for Cochran SR authors, p. 5 of 8 – the summary of methods on p. 4 says that the surveys are anonymous. How will the nonresponse be known if the surveys are anonymous?

Survey design and implementation, p. 5 of 8 - ‘complete a single response for the SR’. It may be more clear to say ‘complete a single survey for the SR’ so as not to confuse with responses to questions on the survey. Will there be a pilot test of the survey?

Items of interest for the surveys include, p. 5 of 8. #5: Reasons for non-use of relevant COS. The sub-bullet stats that the researchers ‘will ask authors whether they will consider incorporating the unused outcomes from COS when completing the SRs…’. This is not an objective of the paper. It seems to be putting the respondents in an awkward position and could lead to socially desirable responses. There may be many reasons why authors cannot add the COS at this time. Perhaps changing to an intention question that asks how likely they would be to add COS in the future would be better. Also, add this component to the objectives if needed. 

Box 1, p. 6 of 8 – Suggest making all bulleted items match grammatically. For example, some bullets have verbs and some do not. For the next to last bullet about Guideline developer: the text in parentheses is confusing. Also it could be made broader than Guideline developer as not all SRs are used for guidelines.

Discussion, p. 6 of 8 – Please add something about expected limitations of the evidence.

Note: The peer review of this protocol is that of the reviewer and does not necessarily represent the official position of the Centers for Disease Control and Prevention.

Is the study design appropriate for the research question?

Partly

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Partly

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

Methods for systematic reviews and efficacy reviews that identify individual evidence-based interventions, HIV prevention.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Saldanha Ian Epidemiology, Johns Hopkins University, Baltimore, Maryland, USA

Competing interests: No competing interests were disclosed.

 20 9 2023

This protocol entitled ‘Current trends, barriers, and facilitators of use of core outcome sets in Cochrane systematic reviews: Protocol’  is describing a protocol of a study that will examine systematic review (SR) protocols and completed reviews for inclusion of core outcome sets (COS) as well as conduct a survey of SR protocol/review authors for barriers and facilitators of using COS.

In general, the protocol is well written and thorough in descriptions. There is some repetition and long sentences that could be reduced. The background does a good job explaining COS and some additional context will help readers not familiar with this concept.

Response: We thank Dr. Sipe for reviewing this manuscript and for providing useful comments. We have edited the manuscript and responded to the comments below.

Background, p. 3 of 8, 1st  paragraph. Term ‘evidence ecosystem’ needs explanation. Seems that time context is needed. The time context will help with understanding the study done in 2019 that is cited in next paragraph. 

Response: We have removed reference to the evidence ecosystem. Instead, we specifically note the other targets of core outcome sets (i.e., for routine care and for registries).

2nd   paragraph – 1st  sentence needs citation.  

Response: We have added references to support the first sentence.

3rd  paragraph – 1st  sentence is long and has a double negative, making it confusing. It also is a conclusion statement without evidence. Encouragement to do the action may be a better way to word the sentence or re-word to become common practice. 2nd  sentence - ‘(i.e., most recent)’ may not be necessary. Last sentence is unclear and Cochrane SRs are personified. It would be the extent in which authors of Cochrane SRs/protocols are considering COS.

Response: We have reworded the first sentence to remove the double-negative phrasing. We have retained the “most recent” in the second sentence because it illustrates the currency of the guidance from Cochrane and because it was released after our prior work, which is discussed in the previous paragraph. We have clarified the last sentence as suggested.

7th paragraph, 1st sentence – the text in parentheses is long and can be shortened ‘(in our previous analyses, we did not survey the authors…)’. Last sentence should ‘both with and beyond Cochrane’ be ‘both within and beyond Cochrane’? If so, this fix is needed for the abstract. 

Response: In the first sentence, we have removed the parenthetical text to be a full sentence and reworded it for clarity. We have made the suggested fix to the last sentence in paragraph 7 as well as in the Abstract.

Methods p. 4 of 8. The design of the study is not stated. It seems to have systematic review components (literature search, inclusion/exclusion criteria, data abstraction) but does not meet all of the requirements for a systematic review. However, it could be considered a rapid review. Regardless, please state what the study design is.

Response: We have added a design descriptor for Objective 1 (“cross-sectional analysis”). The text for Objective 2 includes a design descriptor (“online survey”).

Methods for Objective 1, p. 4 of 8,  eligibility criteria -  Will there be any exclusions?

Response: We have added the following exclusions: “We will exclude SRs of qualitative studies, SRs of methodological topics, and SRs that only address prognosis, diagnostic accuracy, or etiology.”

(4) ‘since’ at the end of the sentence may not be grammatically correct.

Response: We have inserted the word “then” after “since”.

Statistical analyses, p. 4 of 8 – the authors stat that medians and IQRs will be used, however this term is often misused. IQRs are a single number (75th  percentile minus 25th  percentile). So maybe the authors mean ‘interquartile interval’ or more simply 25th  and 75th  percentiles.

Response: We have replaced IQRs with “25 th and 75 th percentiles”.

Eligibility Criteria for Cochran SR authors, p. 5 of 8 – the summary of methods on p. 4 says that the surveys are anonymous. How will the nonresponse be known if the surveys are anonymous?

Response: We have removed the word “anonymous” throughout the manuscript.

Survey design and implementation, p. 5 of 8 - ‘complete a single response for the SR’. It may be more clear to say ‘complete a single survey for the SR’ so as not to confuse with responses to questions on the survey. Will there be a pilot test of the survey?

Response: We have made this change.

Items of interest for the surveys include, p. 5 of 8. #5: Reasons for non-use of relevant COS. The sub-bullet stats that the researchers ‘will ask authors whether they will consider incorporating the unused outcomes from COS when completing the SRs…’. This is not an objective of the paper. It seems to be putting the respondents in an awkward position and could lead to socially desirable responses. There may be many reasons why authors cannot add the COS at this time. Perhaps changing to an intention question that asks how likely they would be to add COS in the future would be better. Also, add this component to the objectives if needed. 

Response: We have modified the sentence to indicate that we will ask authors of such SRs about the factors that would influence their decision to incorporate the unused outcomes from the COS in future SRs.

Box 1, p. 6 of 8 – Suggest making all bulleted items match grammatically. For example, some bullets have verbs and some do not. For the next to last bullet about Guideline developer: the text in parentheses is confusing. Also it could be made broader than Guideline developer as not all SRs are used for guidelines.

Response: We have made the text in the bullet items match grammatically. We have edited the guideline developer text to state “Recommendation of COS use from guideline developer who intended to use the SR findings”.

Discussion, p. 6 of 8 – Please add something about expected limitations of the evidence.

Response: We have added the following limitation to the Discussion section: “A possible limitation of the current study’s findings (if the survey response rate is poor) is that the findings may not be generalizable to all current Cochrane reviewers.”