Effects of evidence-based clinical practice guidelines for breast cancer in health care quality improvements. A second systematic review.

Anggie Ramírez-Morera; Mario Tristán; Jordan Salazar-Vargas; Ana Leonor Rivera-Chavarría

doi:10.12688/f1000research.126126.2

Home Browse Effects of evidence-based clinical practice guidelines for breast...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Systematic Review

Revised

Effects of evidence-based clinical practice guidelines for breast cancer in health care quality improvements. A second systematic review.

[version 2; peer review: 2 approved]

Anggie Ramírez-Morera ^1-3, Mario Tristán¹, Jordan Salazar-Vargas³, Ana Leonor Rivera-Chavarría⁴

PUBLISHED 05 Dec 2022

Author details Author details

¹ Cochrane Central America & Caribbean Spanish, IHCAI Foundation, San José, San José, 10101, Costa Rica
² Universitat Autònoma de Barcelona, Barcelona, Catalunya, 08041, Spain
³ Caja Costarricense de Seguro Social, San José, San José, 10105, Costa Rica
⁴ Instituto Costarricense de Investigación y Enseñanza en Nutrición y Salud, Tres Ríos, Cartago, 42250, Costa Rica

Anggie Ramírez-Morera
Roles: Conceptualization, Data Curation, Formal Analysis, Funding Acquisition, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Mario Tristán
Roles: Conceptualization, Data Curation, Formal Analysis, Funding Acquisition, Methodology, Project Administration, Resources, Software, Writing – Original Draft Preparation, Writing – Review & Editing

Jordan Salazar-Vargas
Roles: Data Curation, Formal Analysis, Funding Acquisition, Resources, Writing – Original Draft Preparation, Writing – Review & Editing

Ana Leonor Rivera-Chavarría
Roles: Data Curation, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Health Services gateway.

Abstract

Background: Traditionally, EB-CPGs have been believed to mainly improve the quality and consistency of health care, but this claim must be conclusively proven. We used the Donabedian three-dimensional model (structure, process, and patient outcomes) to assess improvements in the quality of medical care derived from implementing EB-CPGs. This study corresponds to the second systematic review carried out as a series of studies on different clinical issues that aim to evaluate the effectiveness of the application of the EB-CPG for improving the quality of care.
Methods: We followed the methods described by the Cochrane Handbook and presented a descriptive analysis because of the high heterogeneity found across the included studies. We searched the Cochrane Central Register of Controlled Trials, PubMed, and EBSCO Host databases, as well as the grey literature, between 1990 and April 2021. No language restrictions were applied. Only randomised clinical trials (RCTs) were selected.
Results: Of the total of 364 interventions included in the eleven RCTs evaluated, 11 (3%) were related to healthcare structure, 51 (14%) to the healthcare delivery process and 302 (83%) to patient outcomes. Regarding the impact of using the EB-CPGs, in 303 interventions (83%), there were no significant differences between the control and experimental groups. In 4 interventions (1%), the result favoured the control and intervention groups in 57 of the interventions (16%).
Conclusions: Our study showed that EB-CPGs slightly enhanced the quality of health care in the three dimensions described by Donabedian. Future RCTs should improve their design and methodological rigour by considering the certainty of the evidence supporting the EB-CPGs recommendations. In that context, broader analyses could be performed, having more concise hypotheses for further research.
Registration: PROSPERO CRD42020205594

Keywords

Clinical Practice Guidelines; CPG; effect; health care quality.

Corresponding author: Anggie Ramírez-Morera

Competing interests: No competing interests were disclosed.

Grant information: IHCAI Foundation, the sponsor of the Cochrane Center for Central America and the Spanish-speaking Caribbean, provided the necessary to carry out this review, such as access to databases, full-text articles, and other resources. The funders had no role in study design, data collection and analysis, publication decisions, or manuscript preparation.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2022 Ramírez-Morera A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Ramírez-Morera A, Tristán M, Salazar-Vargas J and Rivera-Chavarría AL. Effects of evidence-based clinical practice guidelines for breast cancer in health care quality improvements. A second systematic review. [version 2; peer review: 2 approved]. F1000Research 2022, 11:1213 (https://doi.org/10.12688/f1000research.126126.2) First published: 25 Oct 2022, 11:1213 (https://doi.org/10.12688/f1000research.126126.1) Latest published: 05 Dec 2022, 11:1213 (https://doi.org/10.12688/f1000research.126126.2)

Revised Amendments from Version 1

We modified the original article in response to the reviewers’ comments:
1. We added breast cancer to the keywords.
2. In the sixth paragraph within the introduction section we added: “and treatment improvement”.

See the authors' detailed response to the review by Ignacio Marín León

Introduction

The emergence of the Evidence-Based Clinical Practice Guidelines (EB-CPGs) in the 1990s has improved decision-making for healthcare personnel and patients with different health conditions (IOM, 1990; Weisz et al., 2007).

Worldwide, significant efforts have been made to develop and implement EB- CPGs based on scientific evidence. The term “evidence-based” means that the recommendations described in the CPGs derive from the best scientific findings and the highest quality of evidence obtained from applying unbiased, transparent, and rigorous methods to support clinical care (Watters, 2008; Linskey, 2010).

EB-CPGs are statements that include recommendations intended to improve the quality and consistency of health care (Woolf et al., 1999; Kwan, 2004) and to assist clinical staff and patients in the decision-making process (Alonso-Coello et al., 2010; IOM, 2011).

This study corresponds to the second systematic review of a series of studies on different clinical subjects, which aim to assess the success of the use of EB-CPGs (Ramírez-Morera et al., 2019). Therefore, we evaluated the effectiveness of the application of EB-CPGs for the improvement of the quality of health care in three dimensions: structure, process, and patient outcome in the management of breast cancer (Donabedian Model, Donabedian, 1988).

Studies measuring the effects of EB-CPGs on the quality of health care have mainly focused on the effects on clinical practice (Lugtenberg et al., 2009), state that some international reviews have demonstrated that most of the studies have resulted in significant improvements to the process of care. However, few studies have focused on the effects of measures on patients’ health outcomes.

We considered for this review breast cancer disease because this is the most common malignancy in women around the world (Ghoncheh et al., 2016). Although survival has improved in the last 30 years mainly because of the implementation of early detection programs and treatment improvement, it still registers 2.3 million new diagnostics in women during 2020 and 685 000 deaths within the same year (WHO, 2020).

Strategies to control and prevent this type of cancer must be a high priority for health policymakers (Ghoncheh et al., 2016). According to the above, hundreds of breast cancer guidelines have been published worldwide to reduce its negative impact on men’s and especially on women’s health.

This review is relevant because there is a growing number of EB-CPGs in different essential areas, and their actual impact on relevant outcomes needs to be assessed. Few systematic reviews evaluate the effect of EBGPC in improving health care; these focus solely on one clinical entity and cover a country or region (Grimshaw et al., 1993; Worrall et al., 1997; Lugtenberg et al., 2009; Ricci-Cabello et al., 2020). For this reason, we reviewed the evidence on the benefits of implementing EB-CPG to improve the quality of care. This review responds to the need for EB-CPG research synthesis on the overall quality of health care delivery.

This systematic review contributes to meeting the need for research synthesis about EB-CPG by assessing the overall quality of health care delivery. Besides, we visualise the need for a systematic review that conclusively demonstrates the pragmatic impact that evidence-based recommendations have on breast cancer patients.

Methods

We conducted a systematic review to identify and analyse the effect of EB-CPGs on health care quality improvement within the Donabedian Model dimensions: structure, process, and results (Donabedian,1988). We followed the Cochrane Handbook methodological recommendations described by Higgins et al. (2022a).

This review is registered at PROSPERO (ID: CRD42020205594).

Study search

The research question was translated into the PICO framework for guiding the study search and the criteria selection (Table 1). We developed a method to incorporate the methodological component of the search strategy combined with selected index and free-text terms.

Table 1. Structure of the clinical question.

Problem	Population	Intervention	Comparison	Outcome
Effects of evidence-based clinical practice guidelines for breast cancer in health care quality improvements	Healthcare professionals involved in breast cancer care	EB-CPGs for the management of breast cancer	Standard care for breast cancer	The impact of EB-CPGs for breast cancer on improving the quality of health care (structure, process, patient outcomes)

We explored the following electronic databases for primary studies: Cochrane Central Register of Controlled Trials (CENTRAL), Cochrane Library, including the Cochrane Effective Practice and Organisation of Care (EPOC) group specialised register, Pubmed, Scopus, EBSCO, Academic Search Complete, CINAHL, Biomedical Reference Collection: Comprehensive, APA PsycInfo, Nursing & Allied Health Collection: Comprehensive, Alt HealthWatch, SPORTDiscus with Full Text, Psychology and Behavioral Sciences Collection, Health Source: Nursing/Academic Edition, Biomedical Reference Collection: Basic, AMED - The Allied and Complementary Medicine Database, Consumer Health Complete, Cochrane Database of Systematic Reviews, Cochrane Methodology Register, Rehabilitation & Sports Medicine Source, AgeLine, Global Health, International Pharmaceutical Abstracts, MasterFILE Premier, Rehabilitation & Sports Medicine Source, LILACS, and Health Technology Assessment Database. We also searched the Science Citation Index and Social Sciences Citation Index for papers that refer to studies included in the review.

The PubMed search strategy was executed in the other databases using the appropriate controlled vocabulary. Searching for other resources included grey literature from different sources and hand searching of those high-yield journals and conference proceedings that have not already been hand searched on behalf of the Cochrane Collaboration.

Authors of relevant papers were contacted regarding any further published or unpublished work. Authors of other reviews in the field of effective professional practice were contacted regarding relevant studies of which they may be aware. We searched for studies published between January 1990 and April 2021. The search strategy was not restricted by language. An advanced search strategy and results are available as extended data, Appendix 1 (Ramírez-Morera et al., 2022).

For the management of bibliographic references of the articles found, the web application “Sciwheel Reference Manager & Generator” was used (Sciwheel, 2022).

Studies selection

The studies found through the search strategy were screened by two reviewers (AR, JS), and discrepancies about study selection were resolved by a third reviewer (MT). Inclusion criteria were: 1. Randomised Clinical Trial (RCT) or cluster-type RCT measuring the impact of using any implementation model versus passive dissemination or no use of the EB-CPG. 2. The studies evaluated the impact on any of the three domains described in the Donabedian model (structure, process, and patient outcomes) for using EB-GPC in treating breast cancer. 3. No language restriction. 4. Published studies from 1990 to 2021.

Data extraction

Three authors (AR, JS, ALR) independently undertook data extraction. They used a modified version of the Cochrane Collaboration EPOC Group “Data Collection Checklist”, employing an electronic datasheet (EPOC, 2019).

To assess the bias risk, we used standard Cochrane methods described in chapters 8, 10 and 23 of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins et al., 2022b; Sterne et al., 2019). In the case of RCTs, bias resulting from several types of systematic errors was assessed according to methods described by Cochrane (Higgins et al., 2022c) and following the RoB2 instrument (Sterne et al., 2019).

All studies deemed eligible for the review were assessed independently by the review authors (AR, MT), and discrepancies were resolved by discussion. A summary of the risk of bias assessment is presented as part of the characteristics of included studies table.

We did not find cluster-type RCTs; then, we did not apply what is stated in chapter 23 of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins et al., 2022d) about assessing cluster RCT following the RoB2 instrument (Sterne et al., 2019).

An analysis of the quality of the evidence related to each of the outcomes was performed using the GRADE approach (Schünemann et al., 2013). We assessed the certainty of the body of evidence for each key outcome as “high”, “moderate”, “low”, or “very low”, using the GRADEpro GDT platform (GRADEpro, 2021).

Data were analysed using Review Manager software, version 5.4 (RevMan, 2020). Revman default templates for data extraction were modified to show the results in a simplified way.

We found very high variability between the measures of effect within the included studies in this review. Then, we decided not to perform a meta-analysis and, therefore, neither to measure statistical heterogeneity.

Results

Study identification and selection

The process describing the analysis of studies retrieved through the systematic search is content in the PRISMA flowchart (Figure 1, Page et al., 2021). We retrieved 25002 studies from database searching and 20 studies were found from additional sources identified. We excluded 15900 duplicated records, and 6072 studies were excluded after screening by title and abstract. We assessed 83 articles at the full-text level, excluding 72 references which did not meet the selection criteria: 51 (71%) were not randomised controlled trials, and 20 (28%) did not evaluate clinical practice guidelines.

Figure 1. PRISMA flowchart of the studies selection process (PRISMA; Page et al., 2021).

* The appendix 1 shows the number of records identified from each database or register searched.

** All records were excluded by the authors. Excluded after screening by Title/Abstract.

*** The records were not randomised controlled trials.

**** The records did not evaluate clinical practice guidelines.

***** They published the results included by Irene et al. (2019), whose study described a broader methodology and reported more detailed results.

From: Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71. For more information, visit: http://www.prisma-statement.org/

Stark et al. (2018) was excluded from this systematic review. They published the results included by Irene et al. (2019), whose study described a broader methodology and more detailed reported results. The list of excluded studies and reasons for exclusion are available as extended data, Appendix 2 (Ramírez-Morera et al., 2022).

Characteristics of the included studies

We included 11 RCTs analysing CPGs for breast cancer (Boekhout et al., 2015; Greenlee et al., 2016; Grunfeld et al., 2011; Hershman et al., 2013; Irene et al., 2019; Klinkhammer-Schalke et al., 2012; Knobf et al., 2016; Maly et al., 2017; Park et al., 2015; Park et al., 2019; Smith-Turchyn et al., 2020). The list of studies selected after screening and assessing the full text is available as extended data, Appendix 3; Ramírez-Morera et al., 2022).

The trials were published between 2011 and 2020, most from 2014 to 2016 (5, 46%). Approximately 55% (6) were carried out in the United States of America. The clinical practice guidelines examined within the selected trials were follow-up (11, 100%) and treatment (4, 36%). When evaluating the outcome categories involving the clinical practice guidelines quoted, most referred to the quality of life (7, 63%). A summary of the characteristics of the included studies (n=11) is available (Table 2). A broader description of these characteristics is available as extended data (Appendix 4; Ramírez-Morera et al., 2022) and the list of the CPGs examined in the included studies (Appendix 5; Ramírez-Morera et al., 2022).

Table 2. Characteristics of all included studies (n=11).

Study characteristics	n (%)	Citation
Country
Canada	3 (27%)	Boekhout et al., 2015; Grunfeld et al., 2011; Smith-Turchyn et al., 2020.
Germany	1 (9%)	Klinkhammer-Schalke et al., 2012.
South Korea	1 (9%)	Park et al., 2015.
USA	6 (55%)	Greenlee et al., 2016; Hershman et al., 2013; Irene et al., 2019; Knobf et al., 2016; Maly et al., 2017; Park et al., 2019.
Publication year
2011-2013	2 (18%)	Grunfeld et al., 2011; Klinkhammer-Schalke et al., 2012.
2014-2016	5 (46%)	Boekhout et al., 2015; Greenlee et al., 2016; Hershman et al., 2013; Knobf et al., 2016; Park et al., 2015.
2017-2020	4 (36%)	Maly et al., 2017; Irene et al., 2019; Park et al., 2019; Smith-Turchyn et al., 2020.
Guideline scope*
Treatment	4 (36%)	Klinkhammer-Schalke et al., 2012; Knobf et al., 2016; Maly et al., 2017; Park et al., 2019.
Follow up	11 (100%)	Boekhout et al., 2015; Greenlee et al., 2016; Grunfeld et al., 2011; Hershman et al., 2013; Irene et al., 2019; Klinkhammer-Schalke et al., 2012.; Knobf et al., 2016; Maly et al., 2017; Park et al., 2015; Park et al., 2019; Smith-Turchyn et al., 2020.
Outcome category*
Exercise	4 (36%)	Boekhout et al., 2015; Knobf et al., 2016; Park et al., 2019; Smith-Turchyn et al., 2020.
Nutrition	3 (27%)	Greenlee et al., 2016; Knobf et al., 2016; Park et al., 2019.
Quality of life	7 (63%)	Grunfeld et al., 2011; Hershman et al., 2013; Irene et al., 2019; Klinkhammer-Schalke et al., 2012.; Maly et al., 2017; Park et al., 2015; Smith-Turchyn et al., 2020.
Treatment	3 (27%)	Boekhout et al., 2015; Klinkhammer-Schalke et al., 2012; Knobf et al., 2016.

* Percentages exceed 100% because the categories are not mutually exclusive (i.e., some studies involved more than one type of guideline scope and more than one outcome category).

Two different studies (Maly et al., 2017; Irene et al., 2019) evaluated the CPG for Fertility preservation for patients with cancer: American Society of Clinical Oncology clinical practice guideline update (Loren et al., 2013). Park et al. (2015) and Smith-Turchyn et al. (2020) analysed the American College of Sports Medicine roundtable on exercise guidelines for cancer survivors (Schmitz et al., 2010).

Additionally, Maly et al. (2017) examined the latest version of the CPG by Khatcheressian et al. (2013), American Society of Clinical Oncology 2006 update of the breast cancer follow-up and management guidelines in the adjuvant setting, previously analysed by Hershman et al. (2013) and Greenlee et al. (2016). They assessed Khatcheressian et al. (2006).

Grunfeld et al. (2011) reported the results until 12 months. Extended results (up to 24 months) for the same study were published by Boekhout et al. (2015). Hershman et al. (2013) and Greenlee et al. (2016) published different results of the same study due to the questionaries or scale utilised for measuring outcomes. The first was more interested in the quality of life and treatment satisfaction, and the second in lifestyle behaviours. We found that Knobf et al. (2016) and Park et al. (2019) published the same study. Knobf et al. (2016) focused on describing the effect of exercise on bone density, while Park et al. (2019) described the results concerning the quality of life.

Risk of bias assessment

We assessed the risk of bias with the RoB2 instrument in the eleven included RCTs following the methods described.

We found that a low risk of bias prevailed (6; 55%) in domain 1: randomisation process. Some concerns occurred in domain 2: deviations from the intended interventions (7; 64%). A low risk of bias was found for domain 3: missing outcome data (10; 91%) and domain 4: measurement of the outcome (8; 73%). Finally, we found some concerns in domain 5: selecting the reported result (7; 64%).

Overall, one study (9%) reported low risk (Irene et al., 2019), some concerns arise from 6 (55%) studies (Grunfeld et al., 2011; Hershman et al., 2013; Boekhout et al., 2015; Park et al., 2015; Greenlee et al., 2016; Smith-Turchyn et al., 2020) and high risk occurred in 4 (36%) studies (Knobf et al., 2016; Klinkhammer-Schalke et al., 2012; Maly et al., 2017; Park et al., 2019). A summary of the results is available in Figure 2.

Figure 2. Analysis of the Risk of Bias (RoB2) for all included studies (n=11).

D1: Randomisation process. D2: Deviations from the intended interventions. D3: Missing outcome data. D4: Measurement of the outcome. D5: Selection of the reported result. Overall risk of bias.

Most of the risk of bias found in the included studies occurred due to the lack of existence or some level of blinding. In some cases, outcome data were not available entirely for all randomised participants. Four studies did not analyse the intention to treat, and some outcome measurement methods were not adequately described. A broader description of the risk of bias assessment is available as extended data in Appendix 6 (Ramírez-Morera et al., 2022).

Quality of evidence assessment

The studies showing the lowest risk of bias (Irene et al., 2019) and one resulting in the highest risk of bias (Park et al., 2019) were chosen to be evaluated with the GRADE methodology and to build a summary of the findings. We decided to grade only four outcomes per study described, including two reporting statistically significant results and two not statistically significant results. Performing a GRADE table provided the rank of possible grades for the certainty of the evidence found in the 362 interventions from the 11 studies included.

The results varied from high to very low certainty of the evidence, according to the GRADE classification. We found several types of systematic errors in the studies: random sequence generation (selection bias, in 4 studies), incomplete outcome data (attrition bias, in 3 studies), and imprecision (observed within all the studies, in 303 outcomes evaluated representing 83% of the total).

We found for the study with the lowest risk of bias (Irene et al., 2019) a range for the certainty of the evidence between high (for fertility-related concerns scale scores ≤3 and improvement with no or low fertility or pregnancy concerns) to moderate certainty of the evidence (for improvement in at least one women’s health issue and 50% decrease in the hot flash score), as described in Table 3a.

Table 3a. Summary of findings: Low risk of bias (Irene et al., 2019).

Patient or population: young breast cancer survivors Setting: web-based, women’s health survivorship care plan (SCP) Intervention: implementation of the recommendations of the CPG Comparison: usual care Outcomes: improve hot flashes, fertility-related concerns, contraception, and vaginal symptoms
Outcomes	Effects		Relative effect (95% CI)	№ of participants (studies)	Certainty of the evidence (GRADE)	Importance
Outcomes	With usual care	With web-based women’s health survivorship care plan (SCP)	Relative effect (95% CI)	№ of participants (studies)	Certainty of the evidence (GRADE)	Importance
Fertility-related concerns scale scores ≤3; assessed with: Reproductive Concerns After Cancer scale (RCAC) follow-up: mean 24 weeks	146 per 1,000	279 per 1,000 (158 to 452)	OR 2.27 (1.10 to 4.84)	182 (1 RCT)	⨁⨁⨁⨁ High^a^,^b^,^h	Important
Improvement with no or low fertility or pregnancy concerns; assessed with: Reproductive Concerns After Cancer scale (RCAC), follow-up: mean 24 weeks	304 per 1,000	533 per 1,000 (329 to 733)	OR 2.61 (1.12 to 6.29)	91 (1 RCT)	⨁⨁⨁⨁ High^a^,^c^,^h	Important
Improvement in at least one women’s health issue, follow-up: mean 24 weeks	573 per 1,000	709 per 1,000 (570 to 820)	OR 1.82 (0.99 to 3.40)	182 (1 RCT)	⨁⨁⨁◯ Moderate^a^,^e	Important
50% decrease in hot flash score; assessed with: Hot Flashes Score, follow-up: mean 24 weeks	552 per 1,000	578 per 1,000 (413 to 729)	OR 1.11 (0.57 to 2.18)	182 (1 RCT)	⨁⨁⨁◯ Moderate^a^,^f	Important

a Risk of bias. Lack of blinding in clinical staff.

b Imprecision. 95% CI: 1.10 to 4.84. p: 0.03. Statistically significant.

c Imprecision. 95% CI: 1.12 to 6.29. p: 0.03. Statistically significant.

e Imprecision. 95% CI: 0.99 to 3.40. p: 0.057. Not statistically significant. Downgraded -1 for imprecision.

f Imprecision. 95% CI: 0.57 to 2.18. p: 0.75. Not statistically significant. Downgraded -1 for imprecision.

h Strong association. OR > 2. Large effect. Upgraded +1.

In the case of Park et al. (2019) study reporting a high risk of bias, we found a range for the certainty of the evidence between low (moderate MET-min/wk and walk MET-min/wk at six months) and very low certainty of the evidence (Moderate MET-min/wk and Walk MET-min/wk at 12 months), as described in Table 3b. Our results after grading the certainty of the evidence for both studies were consistent with the findings from the RoB2 instrument.

Table 3b. Summary of findings: High risk of bias (Park et al., 2019).

Patient or population: peri-menopausal and early postmenopausal female cancer survival Setting: Yale Fitness Intervention Trial Intervention: adherence to the ACS guidelines Comparison: usual care Outcome: improvement of the quality of life (QoL)
Outcomes	Mean effects min/wk ± SD		Absolute effect (95% CI)	№ of participants (studies)	Certainty of the evidence (GRADE)	Importance
Outcomes	With usual care	With adherence to the ACS guidelines	Absolute effect (95% CI)	№ of participants (studies)	Certainty of the evidence (GRADE)	Importance
Moderate MET-min/wk; assessed with: International Physical Activity Questionnaire (IPAQ), follow-up: mean 6 months	33.15 ± 123.87	681.23 ± 127.38	648.08 min/wk (606.091 to 690.069)	138 (1 RCT)	⨁⨁◯◯ Low^a^,^b	Important
Moderate MET-min/wk; assessed with: International Physical Activity Questionnaire (IPAQ), follow-up: mean 12 months	38.19 ± 128.25	115.69 ± 133.59	77.5 min/wk (32.289 to 122.711)	130 (1 RCT)	⨁◯◯◯ Very low^a^,^c	Important
Walk MET-min/wk; assessed with: International Physical Activity Questionnaire (IPAQ), follow-up: mean 6 months	-178.80 ± 124.52	236.63 ± 127.78	415.43 min/wk (373.268 to 457.592)	138 (1 RCT)	⨁⨁◯◯ Low^a^,^d	Important
Walk MET-min/wk; assessed with: International Physical Activity Questionnaire (IPAQ), follow-up: mean 12 months	-126.27 ± 127.75	157.37 ± 132.40	283.64 min/wk (238.718 to 328.562)	130 (1 RCT)	⨁◯◯◯ Very low^a^,^e	Important

a Risk of bias. Lack of allocation concealment. Selection bias. Lack of blinding. Downgraded -2 for risk of bias.

b Imprecision. 95% CI: 606.091 to 690.069. p: 0.0004. Statistically significant.

c Imprecision. 95% CI: 32.289 to 122.711. p: 0.67. Not statistically significant. Downgraded -1 for imprecision.

d Imprecision. 95% CI: 373.268 to 457.592. p: 0.02. Statistically significant.

e Imprecision. 95% CI: 238.718 to 328.562. p: 0.12. Not statistically significant. Downgraded -1 for imprecision.

Assessment of the studies outcomes

The outcomes were grouped in simple relative and absolute numbers. A global estimate of the measurements of the effects included in the studies is lacking because of the significant variability of measuring units combined with the clinical heterogeneity found among the studies included.

There was significant variability in the measurement of the outcomes reported in the studies. Most were continuous, e.g., to assess the quality of life, which corresponds to the dimension of patient outcome. A total of 362 were included in the 11 RCTs evaluated; 11 (3%) corresponded to the health care structure dimension, 51 (14%) interventions to the dimension process and 302 (83%) interventions to the dimension of patient outcomes. A broader description of the main findings by the dimensions evaluated in the included studies is available as extended data, Appendix 7 (Ramírez-Morera et al., 2022).

Regarding the impact of using EB-CPG, we found 303 (83%) interventions with no significant difference between the control and experimental groups. The outcome favoured the control group in 4 (1%). Three outcomes interfered with the patient’s adherence (predictable variables: age, marital status, and hot flashes), as reported by Maly et al. (2017). Also, the fourth outcome informed by Klinkhammer-Schalke et al. (2012) reported rates of therapeutic options for Physiotherapy 16 (experimental group) vs 30 (control group), p: <0.02, at six months. The result favoured the intervention group for 57 interventions (16%) (Table 4).

Table 4. Summary of results for all included studies by dimensions and effect (n=11).

	Citation	Intervention
		Dimension			In favour CPG n (%)	Equal n (%)	In favour control n (%)	Total n (%)
		Structure	Process	Patient outcome	In favour CPG n (%)	Equal n (%)	In favour control n (%)	Total n (%)
1	Boekhout et al., 2015	7 (17%)	20 (49%)	14 (34%)	8 (17%)	33 (83%)	0	41 (100%)
2	Greenlee et al., 2016	0	0	16 (100%)	4 (25%)	12 (75%)	0	16 (100%)
3	Grunfeld et al., 2011	0	6 (25%)	18 (75%)	3 (13%)	21 (87%)	0	24 (100%)
4	Hershman et al., 2013	0	0	27 (100%)	2 (8%)	25 (92%)	0	27 (100%)
5	Irene et al., 2019	0	5 (42%)	7 (58%)	3 (25%)	9 (75%)	0	12 (100%)
6	Klinkhammer-Schalke et al., 2012	0	8 (12%)	60 (88%)	10 (15%)	57 (84%)	1 (1%)	68 (100%)
7	Knobf et al., 2016	0	0	24 (100%)	7 (29%)	17 (71%)	0	24 (100%)
8	Maly et al., 2017	2 (5%)	10 (23%)	32 (72%)	3 (7%)	38 (86%)	3 (7%)	44 (100%)
9	Park et al., 2015	2 (5%)	0	38 (95%)	6 (15%)	34 (85%)	0	40 (100%)
10	Park et al., 2019	0	0	53 (100%)	6 (11%)	47 (89%)	0	53 (100%)
11	Smith-Turchyn et al., 2020	0	0	13 (100%)	3 (23%)	10 (77%)	0	13 (100%)
TOTAL		11 (3%)	51 (14%)	302 (83%)	57 (16%)	303 (83%)	4 (1%)	364 (100%)

Discussion

For more than two decades, governmental and non-governmental institutions have been making economic and methodological efforts to develop more and better quality EB-CPGs, seeking to deal with different issues most healthcare systems face. Such as the ageing population, rising costs motivated by increased demand for quality care, increasingly expensive emerging health technologies, variability in the provision of health by presuming that part of this disparity could cause inadequate care (either overuse or underuse of supplies), and the desire for clinicians and patients to provide and to receive, respectively, the best possible care with measurable clinical effect. However, it still appears that some of these EB-CPGs are far from contributing to an effective, standardised clinical practice based on the best available evidence (Woolf et al., 1999; IOM, 2011; Alonso-Coello et al., 2010). We agree with Woolf et al. (1999) that EB-CPGs that promote proven benefits and discourage ineffective interventions could reduce morbidity and mortality, and improve quality of life, at least for some conditions. EB-CPGs can also improve the consistency of care.

The effects of the recommendations in the interventions included in the 11 RCTs considered in this project were in the structure of medical care (3%) and the care provided (14%); both were the least explored. Surprisingly, patient outcomes were the most evaluated domain (83%), with significant results in 43 of 302 (14%), representing 75% of all significant results.

This fact could lead us to suppose that researchers finally focused on the importance of evaluating the patient health, solely prioritising evaluating the adherence to the CPG (assess the process dimension). They are trying to glimpse more clearly what the use of the CPG represents for patients and not only for clinical staff. As this review considered breast cancer, we could not ignore that there is more social and economic pressure to know the patient outcome.

Grimshaw and Russell (1993) findings described in their systematic review, “Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations”, reported more than 80% significant improvements among the included studies. Contrasting their results, in this second systematic review, we found 57 interventions on breast cancer in favour of the use of EB-CPGs distributed in all the studies included (16% of all the interventions evaluated). Then, compared to our first systematic review, only half of the measures with statistically significant results favour using EB-CPGs (Ramírez-Morera et al., 2019). However, as 75% of these results were found for the dimension of results in patients, we keep optimistic about the finding on incoming reviews an increasingly more significant impact.

Four studies (36%) reported a high risk of bias (Knobf et al., 2016; Klinkhammer-Schalke et al., 2012; Maly et al., 2017; Park et al., 2019), and some concerns arise from 6 (55%) studies (Grunfeld et al., 2011; Hershman et al., 2013; Boekhout et al., 2015; Park et al., 2015; Greenlee et al., 2016; Smith-Turchyn et al., 2020). Because of that, we agree with Ivers et al. (2012) that EB-CPGs must be evaluated, including more remarkable methodological quality studies to provide feedback and corrective measures for clinical practice through audits, promoting improvements in the quality of care.

We also concur with Lugtenberg et al. (2009) about the need to focus on the strength of the recommendations for determining what factors influence the use of the guidelines and the improvement of the results of the patients.

Then we identified the need to perform recommendations distinguishing between the level of certainty of evidence (stratified analysis) as this could have a more significant impact on the results when the best available certainty of evidence recommendation is implemented. We keep in mind that explicit EB-CPG improves clinical practice when introduced within a context of rigorous evaluations (Grimshaw & Russell, 1993; Ricci-Cabello et al., 2020).

Some studies included in the review five reported outcomes in the process area (Boekhout et al., 2015; Grunfeld et al., 2011; Irene et al., 2019; Klinkhammer-Schalke et al., 2012; Maly et al., 2017). All of them described favourable results for the intervention in 14 of 51 measurements (27%). This fact reflects that researchers have continued to endeavour to measure when EB-CPGs should be used or not, but this time to a much lesser extent (14% vs 64%) when compared to the first review.

Lugtenberg et al. (2009) reported that the size of the effects observed in their systematic review varied considerably between the recommendations within the guidelines. We repeated this finding in our study and the previous one (Ramírez-Morera et al., 2019). We found that in many evaluated interventions (303, 83%), the use of EB-CPG did not reflect any impact in any dimension. The approach followed to report the current effectiveness of EB-CPGs remains incomplete (Woolf et al., 1999), and a strategy that captures better results has not yet been found.

EB-CPGs may contribute to improving the quality of healthcare. However, it is still necessary to integrate them with strategies that enhance their use and effect, such as academic and educational visits as part of ongoing training programs (O’Brien et al., 2007). Repeatedly, advocates for EB-CPGs consider their only existence as a magic solution to solve health care problems; however, they ignore other practical actions that should be implemented along with the guidelines (Woolf et al., 1999).

EB-CPG has an essential role when clinicians do not clearly know the appropriate practice and which scientific evidence should support their decisions (Woolf et al., 1999). Then, EB-CPG developers should be vigilant in identifying these needs to help close this information gap and increase the enthusiasm for employing them.

Conclusions

Developing strategies for a more standardised implementation of the EB-CPGs through structured programs within health systems is essential. Improving the awareness of clinical staff about the possibility of enhancing clinical practice and patient outcomes by using evidence-based recommendations with an expected effect is a need.

There is an imbalance between the number of EB-CPGs developed for breast cancer and the number of high-quality studies evaluating their effectiveness. Due to the limited results found on the benefit of using EB-CPG, we must continue investigating the subject. We could gradually structure a more robust hypothesis about the variables influencing this issue.

The variation in the effects found for the recommendations included in the EB-CPGs suggests that it would be helpful to change strategies and focus on the analysis of the limitations of adherence and on designing implementation approaches adapting each recommendation.

In addition, future RCTs should distinguish the levels of certainty of the evidence supporting each recommendation in their evaluations. Researchers should focus on evaluating recommendations expected to have the most significant impact (superior levels of certainty of the evidence: High or Moderate).

More research is necessary to define which factors related to the implementation of EB-CPG and its specific recommendations are essential to predict the application of EB-CPG and, therefore, achieve better patient results.

Implications for the practice

This systematic review aimed to support the development of programs evaluating the effects of EB-CPG on the quality of health care. Also, to provide reliable evidence sustaining the decision-making process related to the production of EB-CPGs.

Even when some of the results of this systematic review were statistically significant, supporting the use of EB-CPG as a tool to improve clinical practice and quality of care, the results of this review need to be interpreted with caution.

The implementation of EB-CPGs must consider the differences in the measures of effect to define customised approaches and specific recommendations within the guideline to enhance health care.

For the adequate implementation of EB-CPGs, it is necessary to consider the possible costs, risks, and benefits and the expected effects derived from EB-CPGs recommendations. The efforts to build EB-CPGs must be complemented with psychosocial strategies promoting health personnel to follow the recommendations of the EB-CPG and evaluate their impact.

Greater methodological rigour in the development of CPGs is needed. It is also required to carry out this process within a standardised formal program in the health systems. Greater credibility could be achieved if recommendations are based on the best available evidence, improving the credibility of their positive effect among health personnel. It could also lead to a more willingness to implement the CPGs and to participate in evaluating their impact.

Implications for the research

This study corresponds to the second systematic review of a series of studies aiming to assess the effect of Evidence-Based Clinical Practice Guidelines (EB-CPGs) on improving health care quality. Our next and last project will investigate Covid-19 disease.

Due to recent research in this field, and the results of this study were not conclusive, more research is necessary to evaluate how EB-CPGs could impact the quality of health care, especially emphasising fewer investigated areas, such as the structure of healthcare services and patient outcomes.

Additionally, the design and methodological rigour applied to future RCTs should improve by considering the certainty of the evidence supporting the EB-CPGs recommendations. Besides, focusing on those with a greater level of evidence (high or moderate) which could lead to determining more clearly the effect that these recommendations have on the quality of health care.

Data (and software) availability

Data availability

Underlying data

All data underlying the results are available as part of the article, and no additional source data are required.

Extended data

Open Science Framework: Extended data for the second SR CPG Breast Cancer. DOI: https://osf.io/6h9pm/?view_only= (Ramírez-Morera et al., 2022).

This project contains the following extended data:

Appendix 1. Advanced search strategy and results.pdf

Appendix 2. List of excluded studies and reasons for exclusion.pdf

Appendix 3. List of selected studies after screening and assessing the full text.pdf

Appendix 4. Characteristics of the included studies.pdf

Appendix 5. List of the CPGs examined in the included studies.pdf

Appendix 6. Risk of Bias 2 assessment.pdf

Appendix 7. Main findings by dimensions of the included studies.pdf

Reporting guidelines

We followed the PRISMA 2020 statement for reporting systematic reviews (Page et al., 2021). We did the PRISMA 2020 checklist and the flow diagram for new systematic reviews, which included searches of databases, registers, and other sources.

Open Science Framework: PRISMA checklist and flow chart for Effects of evidence-based clinical practice guidelines for breast cancer in health care quality improvements. A second systematic review. DOI: https://osf.io/k7f5x/?view_only=7d6b4a63853c43e4b233424ea226e66a.

Author contributions

Anggie Ramírez-Morera: Conceptualisation, Data Curation, Formal Analysis, Funding Acquisition, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing.

Mario Tristán: Conceptualisation, Data Curation, Formal Analysis, Funding Acquisition, Methodology, Resources, Software, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing.

Jordan Salazar-Vargas: Data Curation, Formal Analysis, Funding Acquisition, Resources, Writing – Original Draft Preparation, Writing – Review & Editing.

Ana Leonor Rivera-Chavarría: Data Curation, Writing – Original Draft Preparation, Writing – Review & Editing.

Leading author information

Anggie Ramírez-Morera is PhD candidate Program in Biomedical Research Methodology and Public Health, Universitat Autònoma de Barcelona.

References

Alonso-Coello P, et al.: The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual. Saf. Health Care. 2010; 19(6): e58. PubMed Abstract | Publisher Full Text
Boekhout AH, et al.: A survivorship care plan for breast cancer survivors: extended results of a randomised clinical trial. J. Cancer Surviv. 2015; 9(4): 683–691. PubMed Abstract | Publisher Full Text
Donabedian A: The quality of care. How can it be assessed? J. Am. Med. Assoc. 1988; 260(12): 1743–1748. Publisher Full Text
Cochrane Effective Practice and Organisation of Care (EPOC): Data Collection Checklist. EPOC Resources for review authors, 2017. (updated March 2019). Cochrane.Reference Source
Greenlee H, et al.: Survivorship care plans and adherence to lifestyle recommendations among breast cancer survivors. J. Cancer Surviv. 2016; 10(6): 956–963. PubMed Abstract | Publisher Full Text
Ghoncheh M, Pournamdar Z, Salehiniya H: Incidence and mortality and epidemiology of breast cancer in the world. Asian Pac. J. Cancer Prev. 2016; 17(S3): 43–46. Publisher Full Text
GRADEpro GDT: GRADEpro Guideline Development Tool: [Computer program]. Hamilton (ON): McMaster University (developed by Evidence Prime, Inc.).2021.Reference Source
Grimshaw JM, Russell IT: Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993; 342(8883): 1317–1322. PubMed Abstract | Publisher Full Text
Grunfeld E, et al.: Clinical practice guidelines for the care and treatment of breast cancer: follow-up after treatment for breast cancer (summary of the 2005 update). CMAJ. 2005; 172(10): 1319–1320. PubMed Abstract | Publisher Full Text
Grunfeld E, et al.: Evaluating survivorship care plans: results of a randomised, clinical trial of patients with breast cancer. J. Clin. Oncol. 2011; 29(36): 4755–4762. PubMed Abstract | Publisher Full Text
Hershman DL, et al.: Randomised controlled trial of a clinic-based survivorship intervention following adjuvant therapy in breast cancer survivors. Breast Cancer Res. Treat. 2013; 138(3): 795–806. PubMed Abstract | Publisher Full Text
Higgins JPT, et al.: Cochrane Handbook for Systematic Reviews of Interventions version 6.3. (updated February 2022). Cochrane.2022a.Reference Source
Higgins JPT, et al.:Chapter 6: Choosing effect measures and computing estimates of effect.Higgins JPT, Thomas J, Chandler J, et al. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane;2022b.Reference Source
Higgins JPT, et al.:Chapter 8: Assessing risk of bias in a randomised trial.Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane;2022c.Reference Source
Higgins JPT, et al.:Chapter 23: Including variants on randomised trials.Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane;2022d.Reference Source
Ivers N, et al.: Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst. Rev. 2012; 6: CD000259. PubMed Abstract | Publisher Full Text
Institute of Medicine (US) Committee to Advise the Public Health Service on Clinical Practice Guidelines: Clinical practice guidelines: directions for a new program. Field MJ, Lohr KN, editors.Washington (DC):National Academies Press (US);1990. Publisher Full Text
Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines: Clinical practice guidelines we can trust. Graham R, et al., editors.Washington (DC):National Academies Press (US);2011. Publisher Full Text
Irene S, H., et al.: Efficacy of a web-based women’s health survivorship care plan for young breast cancer survivors: a randomised controlled trial. Breast Cancer Res. Treat. 2019; 176(3): 579–589. PubMed Abstract | Publisher Full Text
Khatcheressian JL, et al.: American Society of Clinical Oncology 2006 update of the breast cancer follow-up and management guidelines in the adjuvant setting. J. Clin. Oncol. 2006; 24(31): 5091–5097. PubMed Abstract | Publisher Full Text
Khatcheressian JL, et al.: Breast cancer follow-up and management after primary treatment: American Society of Clinical Oncology clinical practice guideline update. J. Clin. Oncol. 2013; 31(7): 961–965. PubMed Abstract | Publisher Full Text
Klinkhammer-Schalke M, et al.: Direct improvement of quality of life using a tailored quality of life diagnosis and therapy pathway: randomised trial in 200 women with breast cancer. Br. J. Cancer. 2012; 106(5): 826–838. PubMed Abstract | Publisher Full Text
Knobf MT, et al.: Effect of a randomised controlled exercise trial on bone outcomes: influence of adjuvant endocrine therapy. Breast Cancer Res. Treat. 2016; 155(3): 491–500. PubMed Abstract | Publisher Full Text
Kwan TJ: Do evidence-based guidelines bridge the gaps in research?. Bridging the gaps. Abstracts of the 12th Cochrane Colloquium. 2004 2-6 Oct; Ottawa, Canada. 2004.
Loren AW, et al.: Fertility preservation for patients with cancer: American Society of Clinical Oncology clinical practice guideline update. J. Clin. Oncol. 2013; 31(19): 2500–2510. PubMed Abstract | Publisher Full Text
Linskey ME: Defining excellence in evidence-based medicine clinical practice guidelines. Clin. Neurosurg. 2010; 57: 28–37. PubMed Abstract
Lugtenberg M, Burgers JS, Westert GP: Effects of evidence-based clinical practice guidelines on quality of care: a systematic review. Qual. Saf. Health Care. 2009; 18(5): 385–392. Publisher Full Text
Maly RC, et al.: Randomized Controlled Trial of Survivorship Care Plans Among Low-Income, Predominantly Latina Breast Cancer Survivors. J. Clin. Oncol. 2017; 35(16): 1814–1821. PubMed Abstract | Publisher Full Text
O’Brien MA, et al.: Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst. Rev. 2007; 4: CD000409. PubMed Abstract | Publisher Full Text
Page, et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical Research Ed.). 2021; 372: n71. PubMed Abstract | Publisher Full Text
Park J-H, et al.: The effect of oncologists’ exercise recommendations on the level of exercise and quality of life in survivors of breast and colorectal cancer: A randomised controlled trial. Cancer. 2015; 121(16): 2740–2748. PubMed Abstract | Publisher Full Text
Park S-H, et al.: Adherence to American cancer society guidelines on nutrition and physical activity in female cancer survivors: results from a randomised controlled trial (yale fitness intervention trial). Cancer Nurs. 2019; 42(3): 242–250. PubMed Abstract | Publisher Full Text
Ramírez-Morera A, Tristan M, Vazquez JC: Effects of evidence-based clinical practice guidelines in cardiovascular health care quality improvements: A systematic review [version 3; peer review: 2 approved]. F1000Res. 2019; 8: 1041. PubMed Abstract | Publisher Full Text
Ramírez-Morera A, et al.: Extended data for the second SR CPG Breast Cancer. osf.io/6h9pm.2022.
Review Manager (RevMan): [Computer program]. Version 5.4.1 for Mac. The Cochrane Collaboration;2020.
Ricci-Cabello I, et al.: Adherence to breast cancer guidelines is associated with better survival outcomes: a systematic review and meta-analysis of observational studies in EU countries. BMC Health Serv. Res. 2020; 20(1): 920. PubMed Abstract | Publisher Full Text
Schünemann H, et al.: GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group. 2013.2013.Reference Source
Schmitz KH, et al.: American College of Sports Medicine roundtable on exercise guidelines for cancer survivors. Med. Sci. Sports Exerc. 2010; 42(7): 1409–1426. PubMed Abstract | Publisher Full Text
Sciwheel Reference Manager & Generator: Computer program. MacOS Monterey. London:Sciwheel Ltd.;2022.
Smith-Turchyn J, et al.: Bridging the gap: incorporating exercise evidence into clinical practice in breast cancer care. Support Care Cancer. 2020; 28(2): 897–905. PubMed Abstract | Publisher Full Text
Sterne JAC, et al.: RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ (Clinical Research Ed.). 2019; 366: l4898. PubMed Abstract | Publisher Full Text
Stark SS, et al.: Randomised controlled trial of the effect of a reproductive health survivorship care plan on fertility and pregnancy concerns, vasomotor symptoms, sexual health, and contraception in young breast cancer survivors. Fertil. Steril. 2018; 110(4): e48. Publisher Full Text
Watters W: Defining evidence-based clinical practice guidelines. “AOOS Now”.2008. American Academy of Orthopaedic Surgeons. RESEARCH2.Reference Source Reference Source
Weisz G, et al.: The Emergence of Clinical Practice Guidelines. Milbank Q. 2007; 85(4): 691–727. PubMed Abstract | Publisher Full Text
WHO: Breast Cancer. Cancer prevention. Early Diagnosis and Screening. Geneva:World Health Organization;2020.Reference Source
Woolf SH, et al.: Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ (Clinical Research Ed.). 1999; 318(7182): 527–530. PubMed Abstract | Publisher Full Text | Free Full Text
Worrall G, Chaulk P, Freake D: The effects of clinical practice guidelines on patient outcomes in primary care: a systematic review. Can. Med. Assoc. J. 1997; 156(12): 1705–1712. PubMed Abstract

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 25 Oct 2022

Author details Author details

Jordan Salazar-Vargas
Roles: Data Curation, Formal Analysis, Funding Acquisition, Resources, Writing – Original Draft Preparation, Writing – Review & Editing

Ana Leonor Rivera-Chavarría
Roles: Data Curation, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

IHCAI Foundation, the sponsor of the Cochrane Center for Central America and the Spanish-speaking Caribbean, provided the necessary to carry out this review, such as access to databases, full-text articles, and other resources. The funders had no role in study design, data collection and analysis, publication decisions, or manuscript preparation.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (2)

version 2

Revised

Published: 05 Dec 2022, 11:1213

https://doi.org/10.12688/f1000research.126126.2

version 1

Published: 25 Oct 2022, 11:1213

https://doi.org/10.12688/f1000research.126126.1

© 2022 Ramírez-Morera A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Ramírez-Morera A, Tristán M, Salazar-Vargas J and Rivera-Chavarría AL. Effects of evidence-based clinical practice guidelines for breast cancer in health care quality improvements. A second systematic review. [version 2; peer review: 2 approved]. F1000Research 2022, 11:1213 (https://doi.org/10.12688/f1000research.126126.2)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 2

VERSION 2

PUBLISHED 05 Dec 2022

Revised

Views

Reviewer Report 22 Dec 2022

Ignacio Marín León, Internal Medicine, Virgen del Rocío University Hospital, Seville, Spain

Approved

https://doi.org/10.5256/f1000research.141742.r157384

I agree with the ... Continue reading

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 14 Dec 2022

Wanderley Marques Bernardo, Evidence Based Medicine Department, University of Sao Paulo, Sao Paulo, Brazil; Brazilian Medical Association Guidelines Coordinator, Brazilian Medical Association, Sao Paulo, Brazil

Approved

https://doi.org/10.5256/f1000research.141742.r158163

This is a systematic review of evidence-based guidelines related to breast cancer, and the impact of their use in clinical practice. The method used by the authors achieved all quality items of a systematic review. The authors could not do a meta-analysis due to the heterogeneity of the 11 RCTs selected, and the 364 interventions involved. The main limitation of the systematic review is the expression of results that was mainly descriptive, in spite of some attempt at a quantitative reference to differences between control and experimental groups. But these differences were not associated with specific outcomes. The discussion and conclusions tell us about these limitations and reinforce the importance of considering and focusing on the 3 dimensions when EB guidelines are constructed. This systematic review brings a good model to be reproduced in other clinical subjects of guidelines, considering the adherence and the clinical impacts.

The first quality item of this study is the objective of not only critically evaluating EB guidelines, but mainly evaluating their usefulness measured by clinical outcomes in their implementation.

Analyzing patients with breast cancer as the focus of management, while bringing relevant interventions and outcomes, also brings great variability between conducts, increasing the heterogeneity of the analyses.

In the method, the critical use of three domains (structure, process and result) increases the ability to assess the implementation of the guidelines on the subject.

The search method, selection (eligibility criteria), data extraction, risk of bias assessment, and quality of evidence are fully reproducible and adequate.

There is a limitation in the expression of results due to the already expected heterogeneity between the randomized clinical trials included in the guidelines implementation environment (11 RCTs and more than 300 interventions involved).

The analysis, fundamentally supported by the “outcomes” domain, discourages the use of guidelines, since, in the majority, the results are similar when using or not using a standardized conduct. However, this analysis is not specific about the interventions adopted and measured outcomes, which in addition to not allowing a quantitative expression, does not allow for analyzing the reasons for the absence of differences in the implementation of the recommendations.

And the questions that remain are: is there still any doubt whether the homogenization of medical practice produces benefits and reduces harm to patients? Is this question still open to investigation?

Are the rationale for, and objectives of, the Systematic Review clearly stated?

Yes
Are sufficient details of the methods and analysis provided to allow replication by others?

Yes
Is the statistical analysis and its interpretation appropriate?

Not applicable
Are the conclusions drawn adequately supported by the results presented in the review?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Evidence Based Medicine, Systematic Review and Guidelines

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 25 Oct 2022

Views

Reviewer Report 04 Nov 2022

Ignacio Marín León, Internal Medicine, Virgen del Rocío University Hospital, Seville, Spain

Approved with Reservations

https://doi.org/10.5256/f1000research.138504.r154242

This is an original paper of substantial importance addressing CPG implementation effect. Mostly a very sensible, necessary and well-developed document. In general, the writing is clear and concise, however, needs some editing to avoid unnecessary repetition (i.e. the three domains of Donabedian proposal, and some others). The effect of the implementation of 11 guidelines identified after a rigorous and well-done literature search is presented, as detailed in Methods and Figure 1.

The reviewer has some concerns and suggestions regarding Methods (GRADE use) and Discussion that need to be addressed to improve the manuscript.

An article that is worthy of being indexed.

Major Comments

Methods:
Data extraction, page 4, last paragraph, “An analysis of the quality of the evidence…we assessed the certainty...”: It is unclear how this was done. GRADE is a very labour-intensive and specific methodology, and to perform a GRADE evaluation, the authors have two options: a) simply seek the strength of evidence from the original implemented guideline included in the review, or b) evaluate and critically assess original articles cited in the guidelines reviewed, when they did not use GRADE methodology. This needs clarification on what the authors mean. In addition, readers do not need the authors to explain what GRADE method is.

Results:
Quality of evidence assessment, page 7: This section should be clarified in accordance with previous comments on the use of GRADE.

Discussion:

The structure of this section does not agree with the established "norms" on how the "discussion" section should be drafted. The main findings of the study should first be highlighted in relation to the objective of the study, and in a second step explain the causes of such findings and contrast with similar studies in the available literature. Thus, the large first paragraph could be moved down (and shortened) in the text.
The reviewer does not agree with the statements made in the current third paragraph, in the sense of contrasting the effect on the "process of care" with the effect on "patient outcomes". There are RCTs that relate the implementation of guidelines process of care with the appropriateness and necessity of care for patients' benefits (i.e. see Romero et al. (2005)¹).
Some comment on study weaknesses is missing.

Minor Comments:

Abstract:
It is required that the clinical problem addressed in the study (breast cancer) appears in the abstract and keywords.

Introduction:
In the sixth paragraph: “Although survival has improved…..”: There is a huge debate whether the improved survival depends on the early diagnosis or the improvement of knowledge of the natural history and new treatments of breast cancer (Gøtzsche et al. (2013)², Elmore et al. (2005)³). Thus, some sentences should be complemented with “...of early detection programs and treatment improvement”.

Discussion:
The comments on the usefulness of EB-CPG could be shortened.

Are the rationale for, and objectives of, the Systematic Review clearly stated?

Yes
Are sufficient details of the methods and analysis provided to allow replication by others?

Partly
Is the statistical analysis and its interpretation appropriate?

Yes
Are the conclusions drawn adequately supported by the results presented in the review?

Yes

References

1. Romero A, Alonso C, Marín I, Grimshaw J, et al.: Effectiveness of a Multifactorial Strategy for Implementing Clinical Guidelines on Unstable Angina: Cluster Randomized Trial. Revista Española de Cardiología (English Edition). 2005; 58 (6): 640-648 Publisher Full Text
2. Gøtzsche PC, Jørgensen KJ: Screening for breast cancer with mammography.Cochrane Database Syst Rev. 2013. CD001877 PubMed Abstract | Publisher Full Text
3. Elmore JG, Armstrong K, Lehman CD, Fletcher SW: Screening for breast cancer.JAMA. 2005; 293 (10): 1245-56 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Clinical Epidemiology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 19 Dec 2022

Anggie Ramirez, Universitat Autònoma de Barcelona, Barcelona, 08041, Spain

19 Dec 2022

Author Response

Dear Dr. Ignacio Marin,

We would like to thank you for agreeing to participate in the review of our research. We consider that your contributions are very valuable to ... Continue reading Dear Dr. Ignacio Marin,

We would like to thank you for agreeing to participate in the review of our research. We consider that your contributions are very valuable to improve our work.

Regarding your comments, we describe as follows the research team's position on each of them:

A. Concerning the analysis of the certainty of the evidence, you wrote:
Methods:
Data extraction, page 4, last paragraph, "An analysis of the quality of the evidence…we assessed the certainty...": It is unclear how this was done. GRADE is a very labour-intensive and specific methodology, and to perform a GRADE evaluation, the authors have two options: a) simply seek the strength of evidence from the original implemented guideline included in the review, or b) evaluate and critically assess original articles cited in the guidelines reviewed, when they did not use GRADE methodology. This needs clarification on what the authors mean. In addition, readers do not need the authors to explain what GRADE method is.
Results:
Quality of evidence assessment, page 7: This section should be clarified in accordance with previous comments on the use of GRADE.

Answer to #A:
We agree with you about the options using the GRADE method to develop or adapt a Clinical Practice Guideline. However, we used the GRADE method to assess the certainty of the evidence only for the RCTs that met the inclusion criteria. Therefore, it does not comprise part of the objective and scope of this systematic review to evaluate the evidence included in the CPGs that these studies reviewed.

We wrote as part of the Methods section:
The studies showing the lowest risk of bias (Irene et al., 2019) and one resulting in the highest risk of bias (Park et al., 2019) were chosen to be evaluated with the GRADE methodology and to build a summary of the findings. We decided to grade only four outcomes per study described, including two reporting statistically significant results and two not statistically significant results. Performing a GRADE table provided the rank of possible grades for the certainty of the evidence found in the 362 interventions from the 11 studies included.

Hence, we do not consider it necessary to modify the text in the section "Quality of evidence assessment". Then, we want to clarify that it is part of the objective and scope of this systematic review to evaluate the certainty of the evidence of the RCTs included and not of the EB-CPGs that they evaluated.

B. About the article's Discussion section, you wrote:
B.1
The structure of this section does not agree with the established "norms" on how the "discussion" section should be drafted. The main findings of the study should first be highlighted in relation to the objective of the study, and in a second step explain the causes of such findings and contrast with similar studies in the available literature. Thus, the large first paragraph could be moved down (and shortened) in the text.

Answer to #B.1:
It is relevant to clarify that our article complies with F1000Research guidelines for preparing a Systematic Review article. The journal requests the main body section as follows:
Main Body: The format of the main body of the article is flexible: it should be concise, making it easy to read and review, and presented in a format that is appropriate for the type of study presented. For most Systematic Reviews, the following standard format will be the most appropriate: Introduction, Methods, Results, Conclusions/Discussion.

Also, we consider that the aim of the discussion was achieved because we highlighted our findings and contrasted them with similar research. We must add that we have also complied with the PRISMA 2020 statement checklist (Page et al., 2021[1]). About the discussion, our study meets with the following items:
23a Provide a general interpretation of the results in the context of other evidence.
23b Discuss any limitations of the evidence included in the review.
23c Discuss any limitations of the review processes used.
23d Discuss the implications of the results for practice, policy, and future research.

Therefore, considering that neither of the indications describes an expected order of ideas as you have indicated, the authors alleged it convenient to include this first paragraph to provide a context about the world situation regarding the efforts to develop guidelines and the expectations placed by health caregivers on their effect. Also, to link this finding with the findings in our study.

Consequently, as the first paragraph of this section is congruent with the aim of this systematic review, we consider it appropriate to keep the paragraph unchanged.

B.2 Additionally, regarding your comment about the third paragraph of the discussion section:
The reviewer does not agree with the statements made in the current third paragraph, in the sense of contrasting the effect on the "process of care" with the effect on "patient outcomes". There are RCTs that relate the implementation of guidelines process of care with the appropriateness and necessity of care for patients' benefits (i.e. see Romero et al. (2005)[2]).

We would like to explain the following:
Answer #B.2:
This statement refers to the findings we got from the first systematic review (Ramírez-Morera et al., 2019[3]) that focused on the impact of CPGs in the cardiovascular approach; at that time, we included nine RCTs. Therefore, we decided to compare the results obtained in this second SR to our previous research regarding the number of interventions evaluated according to the dimensions described by Donabedian (1988)[4].

This report corresponds to the result of a Systematic Review that includes only Randomized Controlled Clinical Trials (RCTs) whose objective was to measure the impact that Clinical Practice Guidelines have produced on the quality of health care. This review's authors analyzed the RCT measurements' results according to the framework proposed by Avedis Donabedian^{^[5]} (7 January 1919 – 9 November 2000). Therefore, what has been understood by the implementation of clinical practice guidelines, as defined in the UK glossary published by NICE, adheres to the definition mentioned earlier: "implementation, The process of putting guideline recommendations into practice" (NICE 2014)^{^[6]}. Evaluation components of CPG use, such as clinical auditing, can be of great value. However, clinical auditing has not been disseminated to be available in other countries, predominantly low-income countries.
In 2008, Dutch authors Lugtenberg M, Burgers JS and Westert GP published an excellent systematic review^{^[7]} using the quality of care model by Donabedian. This publication has been very inspiring for the authors of this series of systematic reviews. This review mainly includes RCT studies and some before and after studies done on CPG made in the Netherlands, a country with a long tradition in the design and use of CPG. Some of their results are encouraging, although other aspects were weak, such as the impact on patient's health, which should be considered a very relevant result.

Due to the nature of the data, it was impossible to have a quantitative synthesis, only qualitative. But it did not inhibit us from assessing the certainty of the evidence for the results of the included studies after searching and screening according to the inclusion criteria.
To determine the confidence of the evidence, we have followed Cochrane's recommendations using the tools developed by GRADE. GRADE is a tool that requires knowledge of the essential elements of effective measurement and its potential biases. The authors of this SR are trained to complete this task with adequate competency.

B.3 You also wrote: Some comment on study weaknesses is missing.
Answer #B.3:
We did not write within a specific section of our article what the research's weaknesses are. The weaknesses of the study are described throughout different statements, for example, when describing the risk of bias assessment for the included studies. We also identified many EB-CPGs developed in contrast to the few numbers evaluated through formal RCT-type investigations.

We also wrote about the implications for research:
Due to recent research in this field, and the results of this study were not conclusive, more research is necessary to evaluate how EB-CPGs could impact the quality of health care, especially emphasising fewer investigated areas, such as the structure of healthcare services and patient outcomes.
Additionally, the design and methodological rigour applied to future RCTs should improve by considering the certainty of the evidence supporting the EB-CPGs recommendations. Besides, focusing on those with a greater level of evidence (high or moderate) which could lead to determining more clearly the effect that these recommendations have on the quality of health care.

C. About your minor comments:
C.1 First, about the abstract section you wrote:
Abstract: It is required that the clinical problem addressed in the study (breast cancer) appears in the abstract and keywords.

Answer #C.1:
We agree to add "breast cancer" as a keyword.

C.2 Second, concerning the introduction section, you wrote:
Introduction:
In the sixth paragraph: "Although survival has improved…..": There is a huge debate whether the improved survival depends on the early diagnosis or the improvement of knowledge of the natural history and new treatments of breast cancer (Gøtzsche et al. (2013)[8], Elmore et al. (2005)[9]). Thus, some sentences should be complemented with "...of early detection programs and treatment improvement".

Answer #C.2:
We agree to add the suggested text.

C.3 Third, referring to the discussion section, you wrote:
The comments on the usefulness of EB-CPG could be shortened.

Answer #C.3:
The authors alleged it is convenient to describe the usefulness of EB-CPG because it is congruent with the aim of this systematic review. Therefore, we disagree with the modification suggested.

C.4 Fourth and last, you wrote that there are 'partly' details of the methods and analysis provided to allow replication by others.

Answer #C.4:
Full details on methods and further analyses to allow replication by others can also be accessed through the research protocol, registered at PROSPERO: CRD42020205594.

References

[1] Page et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical Research Ed.), 372, p.n71. [Online]. Available at: doi:10.1136/bmj.n71.

[2] Romero A, Alonso C, Marín I, Grimshaw J, et al.: Effectiveness of a Multifactorial Strategy for Implementing Clinical Guidelines on Unstable Angina: Cluster Randomized Trial. Revista Española de Cardiología (English Edition). 2005; 58 (6): 640-648

[3] Ramírez-Morera, A., Tristan, M. and Vazquez, J. C. (2019). Effects of evidence-based clinical practice guidelines in cardiovascular health care quality improvements: A systematic review [version 3; peer review: 2 approved]. F1000Research 2019, 8:1041[Online]. Available at: https://doi.org/10.12688/f1000research.18865.3

[4] Donabedian, A. (1988). The quality of care. How can it be assessed? The Journal of the American Medical Association, 260 (12), pp.1743–1748. [Online]. Available at: doi:10.1001/jama.260.12.1743.

[5] Donabedian model. (2022, August 8). In Wikipedia. https://en.wikipedia.org/wiki/Donabedian_model

[6] Glossary | Developing NICE guidelines: the manual | Guidance | NICE [WWW Document], 2014 LAST UPDATED 18 January 2022. URL https://www.nice.org.uk/process/pmg20/chapter/glossary

[7] Lugtenberg, M., Burgers, J.S., Westert, G.P., 2009. Effects of evidence-based clinical practice guidelines on quality of care: a systematic review. Qual. Saf. Health Care 18, 385–392. doi:10.1136/qshc.2008.028043

[8] Gøtzsche PC, Jørgensen KJ: Screening for breast cancer with mammography. Cochrane Database Syst Rev. 2013. CD001877

[9] Elmore JG, Armstrong K, Lehman CD, Fletcher SW: Screening for breast cancer. JAMA. 2005; 293 (10): 1245-56
Dear Dr. Ignacio Marin,

We would like to thank you for agreeing to participate in the review of our research. We consider that your contributions are very valuable to improve our work.

Regarding your comments, we describe as follows the research team's position on each of them:

A. Concerning the analysis of the certainty of the evidence, you wrote:
Methods:
Data extraction, page 4, last paragraph, "An analysis of the quality of the evidence…we assessed the certainty...": It is unclear how this was done. GRADE is a very labour-intensive and specific methodology, and to perform a GRADE evaluation, the authors have two options: a) simply seek the strength of evidence from the original implemented guideline included in the review, or b) evaluate and critically assess original articles cited in the guidelines reviewed, when they did not use GRADE methodology. This needs clarification on what the authors mean. In addition, readers do not need the authors to explain what GRADE method is.
Results:
Quality of evidence assessment, page 7: This section should be clarified in accordance with previous comments on the use of GRADE.

Answer to #A:
We agree with you about the options using the GRADE method to develop or adapt a Clinical Practice Guideline. However, we used the GRADE method to assess the certainty of the evidence only for the RCTs that met the inclusion criteria. Therefore, it does not comprise part of the objective and scope of this systematic review to evaluate the evidence included in the CPGs that these studies reviewed.

We wrote as part of the Methods section:
The studies showing the lowest risk of bias (Irene et al., 2019) and one resulting in the highest risk of bias (Park et al., 2019) were chosen to be evaluated with the GRADE methodology and to build a summary of the findings. We decided to grade only four outcomes per study described, including two reporting statistically significant results and two not statistically significant results. Performing a GRADE table provided the rank of possible grades for the certainty of the evidence found in the 362 interventions from the 11 studies included.

Hence, we do not consider it necessary to modify the text in the section "Quality of evidence assessment". Then, we want to clarify that it is part of the objective and scope of this systematic review to evaluate the certainty of the evidence of the RCTs included and not of the EB-CPGs that they evaluated.

B. About the article's Discussion section, you wrote:
B.1
The structure of this section does not agree with the established "norms" on how the "discussion" section should be drafted. The main findings of the study should first be highlighted in relation to the objective of the study, and in a second step explain the causes of such findings and contrast with similar studies in the available literature. Thus, the large first paragraph could be moved down (and shortened) in the text.

Answer to #B.1:
It is relevant to clarify that our article complies with F1000Research guidelines for preparing a Systematic Review article. The journal requests the main body section as follows:
Main Body: The format of the main body of the article is flexible: it should be concise, making it easy to read and review, and presented in a format that is appropriate for the type of study presented. For most Systematic Reviews, the following standard format will be the most appropriate: Introduction, Methods, Results, Conclusions/Discussion.

Also, we consider that the aim of the discussion was achieved because we highlighted our findings and contrasted them with similar research. We must add that we have also complied with the PRISMA 2020 statement checklist (Page et al., 2021[1]). About the discussion, our study meets with the following items:
23a Provide a general interpretation of the results in the context of other evidence.
23b Discuss any limitations of the evidence included in the review.
23c Discuss any limitations of the review processes used.
23d Discuss the implications of the results for practice, policy, and future research.

Therefore, considering that neither of the indications describes an expected order of ideas as you have indicated, the authors alleged it convenient to include this first paragraph to provide a context about the world situation regarding the efforts to develop guidelines and the expectations placed by health caregivers on their effect. Also, to link this finding with the findings in our study.

Consequently, as the first paragraph of this section is congruent with the aim of this systematic review, we consider it appropriate to keep the paragraph unchanged.

B.2 Additionally, regarding your comment about the third paragraph of the discussion section:
The reviewer does not agree with the statements made in the current third paragraph, in the sense of contrasting the effect on the "process of care" with the effect on "patient outcomes". There are RCTs that relate the implementation of guidelines process of care with the appropriateness and necessity of care for patients' benefits (i.e. see Romero et al. (2005)[2]).

We would like to explain the following:
Answer #B.2:
This statement refers to the findings we got from the first systematic review (Ramírez-Morera et al., 2019[3]) that focused on the impact of CPGs in the cardiovascular approach; at that time, we included nine RCTs. Therefore, we decided to compare the results obtained in this second SR to our previous research regarding the number of interventions evaluated according to the dimensions described by Donabedian (1988)[4].

This report corresponds to the result of a Systematic Review that includes only Randomized Controlled Clinical Trials (RCTs) whose objective was to measure the impact that Clinical Practice Guidelines have produced on the quality of health care. This review's authors analyzed the RCT measurements' results according to the framework proposed by Avedis Donabedian^{^[5]} (7 January 1919 – 9 November 2000). Therefore, what has been understood by the implementation of clinical practice guidelines, as defined in the UK glossary published by NICE, adheres to the definition mentioned earlier: "implementation, The process of putting guideline recommendations into practice" (NICE 2014)^{^[6]}. Evaluation components of CPG use, such as clinical auditing, can be of great value. However, clinical auditing has not been disseminated to be available in other countries, predominantly low-income countries.
In 2008, Dutch authors Lugtenberg M, Burgers JS and Westert GP published an excellent systematic review^{^[7]} using the quality of care model by Donabedian. This publication has been very inspiring for the authors of this series of systematic reviews. This review mainly includes RCT studies and some before and after studies done on CPG made in the Netherlands, a country with a long tradition in the design and use of CPG. Some of their results are encouraging, although other aspects were weak, such as the impact on patient's health, which should be considered a very relevant result.

Due to the nature of the data, it was impossible to have a quantitative synthesis, only qualitative. But it did not inhibit us from assessing the certainty of the evidence for the results of the included studies after searching and screening according to the inclusion criteria.
To determine the confidence of the evidence, we have followed Cochrane's recommendations using the tools developed by GRADE. GRADE is a tool that requires knowledge of the essential elements of effective measurement and its potential biases. The authors of this SR are trained to complete this task with adequate competency.

B.3 You also wrote: Some comment on study weaknesses is missing.
Answer #B.3:
We did not write within a specific section of our article what the research's weaknesses are. The weaknesses of the study are described throughout different statements, for example, when describing the risk of bias assessment for the included studies. We also identified many EB-CPGs developed in contrast to the few numbers evaluated through formal RCT-type investigations.

We also wrote about the implications for research:
Due to recent research in this field, and the results of this study were not conclusive, more research is necessary to evaluate how EB-CPGs could impact the quality of health care, especially emphasising fewer investigated areas, such as the structure of healthcare services and patient outcomes.
Additionally, the design and methodological rigour applied to future RCTs should improve by considering the certainty of the evidence supporting the EB-CPGs recommendations. Besides, focusing on those with a greater level of evidence (high or moderate) which could lead to determining more clearly the effect that these recommendations have on the quality of health care.

C. About your minor comments:
C.1 First, about the abstract section you wrote:
Abstract: It is required that the clinical problem addressed in the study (breast cancer) appears in the abstract and keywords.

Answer #C.1:
We agree to add "breast cancer" as a keyword.

C.2 Second, concerning the introduction section, you wrote:
Introduction:
In the sixth paragraph: "Although survival has improved…..": There is a huge debate whether the improved survival depends on the early diagnosis or the improvement of knowledge of the natural history and new treatments of breast cancer (Gøtzsche et al. (2013)[8], Elmore et al. (2005)[9]). Thus, some sentences should be complemented with "...of early detection programs and treatment improvement".

Answer #C.2:
We agree to add the suggested text.

C.3 Third, referring to the discussion section, you wrote:
The comments on the usefulness of EB-CPG could be shortened.

Answer #C.3:
The authors alleged it is convenient to describe the usefulness of EB-CPG because it is congruent with the aim of this systematic review. Therefore, we disagree with the modification suggested.

C.4 Fourth and last, you wrote that there are 'partly' details of the methods and analysis provided to allow replication by others.

Answer #C.4:
Full details on methods and further analyses to allow replication by others can also be accessed through the research protocol, registered at PROSPERO: CRD42020205594.

References

[1] Page et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical Research Ed.), 372, p.n71. [Online]. Available at: doi:10.1136/bmj.n71.

[2] Romero A, Alonso C, Marín I, Grimshaw J, et al.: Effectiveness of a Multifactorial Strategy for Implementing Clinical Guidelines on Unstable Angina: Cluster Randomized Trial. Revista Española de Cardiología (English Edition). 2005; 58 (6): 640-648

[3] Ramírez-Morera, A., Tristan, M. and Vazquez, J. C. (2019). Effects of evidence-based clinical practice guidelines in cardiovascular health care quality improvements: A systematic review [version 3; peer review: 2 approved]. F1000Research 2019, 8:1041[Online]. Available at: https://doi.org/10.12688/f1000research.18865.3

[4] Donabedian, A. (1988). The quality of care. How can it be assessed? The Journal of the American Medical Association, 260 (12), pp.1743–1748. [Online]. Available at: doi:10.1001/jama.260.12.1743.

[5] Donabedian model. (2022, August 8). In Wikipedia. https://en.wikipedia.org/wiki/Donabedian_model

[6] Glossary | Developing NICE guidelines: the manual | Guidance | NICE [WWW Document], 2014 LAST UPDATED 18 January 2022. URL https://www.nice.org.uk/process/pmg20/chapter/glossary

[7] Lugtenberg, M., Burgers, J.S., Westert, G.P., 2009. Effects of evidence-based clinical practice guidelines on quality of care: a systematic review. Qual. Saf. Health Care 18, 385–392. doi:10.1136/qshc.2008.028043

[8] Gøtzsche PC, Jørgensen KJ: Screening for breast cancer with mammography. Cochrane Database Syst Rev. 2013. CD001877

[9] Elmore JG, Armstrong K, Lehman CD, Fletcher SW: Screening for breast cancer. JAMA. 2005; 293 (10): 1245-56
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 19 Dec 2022

Anggie Ramirez, Universitat Autònoma de Barcelona, Barcelona, 08041, Spain

19 Dec 2022

Author Response

Dear Dr. Ignacio Marin,

We would like to thank you for agreeing to participate in the review of our research. We consider that your contributions are very valuable to ... Continue reading Dear Dr. Ignacio Marin,

We would like to thank you for agreeing to participate in the review of our research. We consider that your contributions are very valuable to improve our work.

Regarding your comments, we describe as follows the research team's position on each of them:

A. Concerning the analysis of the certainty of the evidence, you wrote:
Methods:
Data extraction, page 4, last paragraph, "An analysis of the quality of the evidence…we assessed the certainty...": It is unclear how this was done. GRADE is a very labour-intensive and specific methodology, and to perform a GRADE evaluation, the authors have two options: a) simply seek the strength of evidence from the original implemented guideline included in the review, or b) evaluate and critically assess original articles cited in the guidelines reviewed, when they did not use GRADE methodology. This needs clarification on what the authors mean. In addition, readers do not need the authors to explain what GRADE method is.
Results:
Quality of evidence assessment, page 7: This section should be clarified in accordance with previous comments on the use of GRADE.

Answer to #A:
We agree with you about the options using the GRADE method to develop or adapt a Clinical Practice Guideline. However, we used the GRADE method to assess the certainty of the evidence only for the RCTs that met the inclusion criteria. Therefore, it does not comprise part of the objective and scope of this systematic review to evaluate the evidence included in the CPGs that these studies reviewed.

We wrote as part of the Methods section:
The studies showing the lowest risk of bias (Irene et al., 2019) and one resulting in the highest risk of bias (Park et al., 2019) were chosen to be evaluated with the GRADE methodology and to build a summary of the findings. We decided to grade only four outcomes per study described, including two reporting statistically significant results and two not statistically significant results. Performing a GRADE table provided the rank of possible grades for the certainty of the evidence found in the 362 interventions from the 11 studies included.

Hence, we do not consider it necessary to modify the text in the section "Quality of evidence assessment". Then, we want to clarify that it is part of the objective and scope of this systematic review to evaluate the certainty of the evidence of the RCTs included and not of the EB-CPGs that they evaluated.

B. About the article's Discussion section, you wrote:
B.1
The structure of this section does not agree with the established "norms" on how the "discussion" section should be drafted. The main findings of the study should first be highlighted in relation to the objective of the study, and in a second step explain the causes of such findings and contrast with similar studies in the available literature. Thus, the large first paragraph could be moved down (and shortened) in the text.

Answer to #B.1:
It is relevant to clarify that our article complies with F1000Research guidelines for preparing a Systematic Review article. The journal requests the main body section as follows:
Main Body: The format of the main body of the article is flexible: it should be concise, making it easy to read and review, and presented in a format that is appropriate for the type of study presented. For most Systematic Reviews, the following standard format will be the most appropriate: Introduction, Methods, Results, Conclusions/Discussion.

Also, we consider that the aim of the discussion was achieved because we highlighted our findings and contrasted them with similar research. We must add that we have also complied with the PRISMA 2020 statement checklist (Page et al., 2021[1]). About the discussion, our study meets with the following items:
23a Provide a general interpretation of the results in the context of other evidence.
23b Discuss any limitations of the evidence included in the review.
23c Discuss any limitations of the review processes used.
23d Discuss the implications of the results for practice, policy, and future research.

Therefore, considering that neither of the indications describes an expected order of ideas as you have indicated, the authors alleged it convenient to include this first paragraph to provide a context about the world situation regarding the efforts to develop guidelines and the expectations placed by health caregivers on their effect. Also, to link this finding with the findings in our study.

Consequently, as the first paragraph of this section is congruent with the aim of this systematic review, we consider it appropriate to keep the paragraph unchanged.

B.2 Additionally, regarding your comment about the third paragraph of the discussion section:
The reviewer does not agree with the statements made in the current third paragraph, in the sense of contrasting the effect on the "process of care" with the effect on "patient outcomes". There are RCTs that relate the implementation of guidelines process of care with the appropriateness and necessity of care for patients' benefits (i.e. see Romero et al. (2005)[2]).

We would like to explain the following:
Answer #B.2:
This statement refers to the findings we got from the first systematic review (Ramírez-Morera et al., 2019[3]) that focused on the impact of CPGs in the cardiovascular approach; at that time, we included nine RCTs. Therefore, we decided to compare the results obtained in this second SR to our previous research regarding the number of interventions evaluated according to the dimensions described by Donabedian (1988)[4].

This report corresponds to the result of a Systematic Review that includes only Randomized Controlled Clinical Trials (RCTs) whose objective was to measure the impact that Clinical Practice Guidelines have produced on the quality of health care. This review's authors analyzed the RCT measurements' results according to the framework proposed by Avedis Donabedian^{^[5]} (7 January 1919 – 9 November 2000). Therefore, what has been understood by the implementation of clinical practice guidelines, as defined in the UK glossary published by NICE, adheres to the definition mentioned earlier: "implementation, The process of putting guideline recommendations into practice" (NICE 2014)^{^[6]}. Evaluation components of CPG use, such as clinical auditing, can be of great value. However, clinical auditing has not been disseminated to be available in other countries, predominantly low-income countries.
In 2008, Dutch authors Lugtenberg M, Burgers JS and Westert GP published an excellent systematic review^{^[7]} using the quality of care model by Donabedian. This publication has been very inspiring for the authors of this series of systematic reviews. This review mainly includes RCT studies and some before and after studies done on CPG made in the Netherlands, a country with a long tradition in the design and use of CPG. Some of their results are encouraging, although other aspects were weak, such as the impact on patient's health, which should be considered a very relevant result.

Due to the nature of the data, it was impossible to have a quantitative synthesis, only qualitative. But it did not inhibit us from assessing the certainty of the evidence for the results of the included studies after searching and screening according to the inclusion criteria.
To determine the confidence of the evidence, we have followed Cochrane's recommendations using the tools developed by GRADE. GRADE is a tool that requires knowledge of the essential elements of effective measurement and its potential biases. The authors of this SR are trained to complete this task with adequate competency.

B.3 You also wrote: Some comment on study weaknesses is missing.
Answer #B.3:
We did not write within a specific section of our article what the research's weaknesses are. The weaknesses of the study are described throughout different statements, for example, when describing the risk of bias assessment for the included studies. We also identified many EB-CPGs developed in contrast to the few numbers evaluated through formal RCT-type investigations.

We also wrote about the implications for research:
Due to recent research in this field, and the results of this study were not conclusive, more research is necessary to evaluate how EB-CPGs could impact the quality of health care, especially emphasising fewer investigated areas, such as the structure of healthcare services and patient outcomes.
Additionally, the design and methodological rigour applied to future RCTs should improve by considering the certainty of the evidence supporting the EB-CPGs recommendations. Besides, focusing on those with a greater level of evidence (high or moderate) which could lead to determining more clearly the effect that these recommendations have on the quality of health care.

C. About your minor comments:
C.1 First, about the abstract section you wrote:
Abstract: It is required that the clinical problem addressed in the study (breast cancer) appears in the abstract and keywords.

Answer #C.1:
We agree to add "breast cancer" as a keyword.

C.2 Second, concerning the introduction section, you wrote:
Introduction:
In the sixth paragraph: "Although survival has improved…..": There is a huge debate whether the improved survival depends on the early diagnosis or the improvement of knowledge of the natural history and new treatments of breast cancer (Gøtzsche et al. (2013)[8], Elmore et al. (2005)[9]). Thus, some sentences should be complemented with "...of early detection programs and treatment improvement".

Answer #C.2:
We agree to add the suggested text.

C.3 Third, referring to the discussion section, you wrote:
The comments on the usefulness of EB-CPG could be shortened.

Answer #C.3:
The authors alleged it is convenient to describe the usefulness of EB-CPG because it is congruent with the aim of this systematic review. Therefore, we disagree with the modification suggested.

C.4 Fourth and last, you wrote that there are 'partly' details of the methods and analysis provided to allow replication by others.

Answer #C.4:
Full details on methods and further analyses to allow replication by others can also be accessed through the research protocol, registered at PROSPERO: CRD42020205594.

References

[1] Page et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical Research Ed.), 372, p.n71. [Online]. Available at: doi:10.1136/bmj.n71.

[2] Romero A, Alonso C, Marín I, Grimshaw J, et al.: Effectiveness of a Multifactorial Strategy for Implementing Clinical Guidelines on Unstable Angina: Cluster Randomized Trial. Revista Española de Cardiología (English Edition). 2005; 58 (6): 640-648

[3] Ramírez-Morera, A., Tristan, M. and Vazquez, J. C. (2019). Effects of evidence-based clinical practice guidelines in cardiovascular health care quality improvements: A systematic review [version 3; peer review: 2 approved]. F1000Research 2019, 8:1041[Online]. Available at: https://doi.org/10.12688/f1000research.18865.3

[4] Donabedian, A. (1988). The quality of care. How can it be assessed? The Journal of the American Medical Association, 260 (12), pp.1743–1748. [Online]. Available at: doi:10.1001/jama.260.12.1743.

[5] Donabedian model. (2022, August 8). In Wikipedia. https://en.wikipedia.org/wiki/Donabedian_model

[6] Glossary | Developing NICE guidelines: the manual | Guidance | NICE [WWW Document], 2014 LAST UPDATED 18 January 2022. URL https://www.nice.org.uk/process/pmg20/chapter/glossary

[7] Lugtenberg, M., Burgers, J.S., Westert, G.P., 2009. Effects of evidence-based clinical practice guidelines on quality of care: a systematic review. Qual. Saf. Health Care 18, 385–392. doi:10.1136/qshc.2008.028043

[8] Gøtzsche PC, Jørgensen KJ: Screening for breast cancer with mammography. Cochrane Database Syst Rev. 2013. CD001877

[9] Elmore JG, Armstrong K, Lehman CD, Fletcher SW: Screening for breast cancer. JAMA. 2005; 293 (10): 1245-56
Dear Dr. Ignacio Marin,

We would like to thank you for agreeing to participate in the review of our research. We consider that your contributions are very valuable to improve our work.

Regarding your comments, we describe as follows the research team's position on each of them:

A. Concerning the analysis of the certainty of the evidence, you wrote:
Methods:
Data extraction, page 4, last paragraph, "An analysis of the quality of the evidence…we assessed the certainty...": It is unclear how this was done. GRADE is a very labour-intensive and specific methodology, and to perform a GRADE evaluation, the authors have two options: a) simply seek the strength of evidence from the original implemented guideline included in the review, or b) evaluate and critically assess original articles cited in the guidelines reviewed, when they did not use GRADE methodology. This needs clarification on what the authors mean. In addition, readers do not need the authors to explain what GRADE method is.
Results:
Quality of evidence assessment, page 7: This section should be clarified in accordance with previous comments on the use of GRADE.

Answer to #A:
We agree with you about the options using the GRADE method to develop or adapt a Clinical Practice Guideline. However, we used the GRADE method to assess the certainty of the evidence only for the RCTs that met the inclusion criteria. Therefore, it does not comprise part of the objective and scope of this systematic review to evaluate the evidence included in the CPGs that these studies reviewed.

We wrote as part of the Methods section:
The studies showing the lowest risk of bias (Irene et al., 2019) and one resulting in the highest risk of bias (Park et al., 2019) were chosen to be evaluated with the GRADE methodology and to build a summary of the findings. We decided to grade only four outcomes per study described, including two reporting statistically significant results and two not statistically significant results. Performing a GRADE table provided the rank of possible grades for the certainty of the evidence found in the 362 interventions from the 11 studies included.

Hence, we do not consider it necessary to modify the text in the section "Quality of evidence assessment". Then, we want to clarify that it is part of the objective and scope of this systematic review to evaluate the certainty of the evidence of the RCTs included and not of the EB-CPGs that they evaluated.

B. About the article's Discussion section, you wrote:
B.1
The structure of this section does not agree with the established "norms" on how the "discussion" section should be drafted. The main findings of the study should first be highlighted in relation to the objective of the study, and in a second step explain the causes of such findings and contrast with similar studies in the available literature. Thus, the large first paragraph could be moved down (and shortened) in the text.

Answer to #B.1:
It is relevant to clarify that our article complies with F1000Research guidelines for preparing a Systematic Review article. The journal requests the main body section as follows:
Main Body: The format of the main body of the article is flexible: it should be concise, making it easy to read and review, and presented in a format that is appropriate for the type of study presented. For most Systematic Reviews, the following standard format will be the most appropriate: Introduction, Methods, Results, Conclusions/Discussion.

Also, we consider that the aim of the discussion was achieved because we highlighted our findings and contrasted them with similar research. We must add that we have also complied with the PRISMA 2020 statement checklist (Page et al., 2021[1]). About the discussion, our study meets with the following items:
23a Provide a general interpretation of the results in the context of other evidence.
23b Discuss any limitations of the evidence included in the review.
23c Discuss any limitations of the review processes used.
23d Discuss the implications of the results for practice, policy, and future research.

Therefore, considering that neither of the indications describes an expected order of ideas as you have indicated, the authors alleged it convenient to include this first paragraph to provide a context about the world situation regarding the efforts to develop guidelines and the expectations placed by health caregivers on their effect. Also, to link this finding with the findings in our study.

Consequently, as the first paragraph of this section is congruent with the aim of this systematic review, we consider it appropriate to keep the paragraph unchanged.

B.2 Additionally, regarding your comment about the third paragraph of the discussion section:
The reviewer does not agree with the statements made in the current third paragraph, in the sense of contrasting the effect on the "process of care" with the effect on "patient outcomes". There are RCTs that relate the implementation of guidelines process of care with the appropriateness and necessity of care for patients' benefits (i.e. see Romero et al. (2005)[2]).

We would like to explain the following:
Answer #B.2:
This statement refers to the findings we got from the first systematic review (Ramírez-Morera et al., 2019[3]) that focused on the impact of CPGs in the cardiovascular approach; at that time, we included nine RCTs. Therefore, we decided to compare the results obtained in this second SR to our previous research regarding the number of interventions evaluated according to the dimensions described by Donabedian (1988)[4].

This report corresponds to the result of a Systematic Review that includes only Randomized Controlled Clinical Trials (RCTs) whose objective was to measure the impact that Clinical Practice Guidelines have produced on the quality of health care. This review's authors analyzed the RCT measurements' results according to the framework proposed by Avedis Donabedian^{^[5]} (7 January 1919 – 9 November 2000). Therefore, what has been understood by the implementation of clinical practice guidelines, as defined in the UK glossary published by NICE, adheres to the definition mentioned earlier: "implementation, The process of putting guideline recommendations into practice" (NICE 2014)^{^[6]}. Evaluation components of CPG use, such as clinical auditing, can be of great value. However, clinical auditing has not been disseminated to be available in other countries, predominantly low-income countries.
In 2008, Dutch authors Lugtenberg M, Burgers JS and Westert GP published an excellent systematic review^{^[7]} using the quality of care model by Donabedian. This publication has been very inspiring for the authors of this series of systematic reviews. This review mainly includes RCT studies and some before and after studies done on CPG made in the Netherlands, a country with a long tradition in the design and use of CPG. Some of their results are encouraging, although other aspects were weak, such as the impact on patient's health, which should be considered a very relevant result.

Due to the nature of the data, it was impossible to have a quantitative synthesis, only qualitative. But it did not inhibit us from assessing the certainty of the evidence for the results of the included studies after searching and screening according to the inclusion criteria.
To determine the confidence of the evidence, we have followed Cochrane's recommendations using the tools developed by GRADE. GRADE is a tool that requires knowledge of the essential elements of effective measurement and its potential biases. The authors of this SR are trained to complete this task with adequate competency.

B.3 You also wrote: Some comment on study weaknesses is missing.
Answer #B.3:
We did not write within a specific section of our article what the research's weaknesses are. The weaknesses of the study are described throughout different statements, for example, when describing the risk of bias assessment for the included studies. We also identified many EB-CPGs developed in contrast to the few numbers evaluated through formal RCT-type investigations.

We also wrote about the implications for research:
Due to recent research in this field, and the results of this study were not conclusive, more research is necessary to evaluate how EB-CPGs could impact the quality of health care, especially emphasising fewer investigated areas, such as the structure of healthcare services and patient outcomes.
Additionally, the design and methodological rigour applied to future RCTs should improve by considering the certainty of the evidence supporting the EB-CPGs recommendations. Besides, focusing on those with a greater level of evidence (high or moderate) which could lead to determining more clearly the effect that these recommendations have on the quality of health care.

C. About your minor comments:
C.1 First, about the abstract section you wrote:
Abstract: It is required that the clinical problem addressed in the study (breast cancer) appears in the abstract and keywords.

Answer #C.1:
We agree to add "breast cancer" as a keyword.

C.2 Second, concerning the introduction section, you wrote:
Introduction:
In the sixth paragraph: "Although survival has improved…..": There is a huge debate whether the improved survival depends on the early diagnosis or the improvement of knowledge of the natural history and new treatments of breast cancer (Gøtzsche et al. (2013)[8], Elmore et al. (2005)[9]). Thus, some sentences should be complemented with "...of early detection programs and treatment improvement".

Answer #C.2:
We agree to add the suggested text.

C.3 Third, referring to the discussion section, you wrote:
The comments on the usefulness of EB-CPG could be shortened.

Answer #C.3:
The authors alleged it is convenient to describe the usefulness of EB-CPG because it is congruent with the aim of this systematic review. Therefore, we disagree with the modification suggested.

C.4 Fourth and last, you wrote that there are 'partly' details of the methods and analysis provided to allow replication by others.

Answer #C.4:
Full details on methods and further analyses to allow replication by others can also be accessed through the research protocol, registered at PROSPERO: CRD42020205594.

References

[1] Page et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical Research Ed.), 372, p.n71. [Online]. Available at: doi:10.1136/bmj.n71.

[2] Romero A, Alonso C, Marín I, Grimshaw J, et al.: Effectiveness of a Multifactorial Strategy for Implementing Clinical Guidelines on Unstable Angina: Cluster Randomized Trial. Revista Española de Cardiología (English Edition). 2005; 58 (6): 640-648

[3] Ramírez-Morera, A., Tristan, M. and Vazquez, J. C. (2019). Effects of evidence-based clinical practice guidelines in cardiovascular health care quality improvements: A systematic review [version 3; peer review: 2 approved]. F1000Research 2019, 8:1041[Online]. Available at: https://doi.org/10.12688/f1000research.18865.3

[4] Donabedian, A. (1988). The quality of care. How can it be assessed? The Journal of the American Medical Association, 260 (12), pp.1743–1748. [Online]. Available at: doi:10.1001/jama.260.12.1743.

[5] Donabedian model. (2022, August 8). In Wikipedia. https://en.wikipedia.org/wiki/Donabedian_model

[6] Glossary | Developing NICE guidelines: the manual | Guidance | NICE [WWW Document], 2014 LAST UPDATED 18 January 2022. URL https://www.nice.org.uk/process/pmg20/chapter/glossary

[7] Lugtenberg, M., Burgers, J.S., Westert, G.P., 2009. Effects of evidence-based clinical practice guidelines on quality of care: a systematic review. Qual. Saf. Health Care 18, 385–392. doi:10.1136/qshc.2008.028043

[8] Gøtzsche PC, Jørgensen KJ: Screening for breast cancer with mammography. Cochrane Database Syst Rev. 2013. CD001877

[9] Elmore JG, Armstrong K, Lehman CD, Fletcher SW: Screening for breast cancer. JAMA. 2005; 293 (10): 1245-56
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 25 Oct 2022

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 2 (revision) 05 Dec 22	read	read
Version 1 25 Oct 22	read

Ignacio Marín León, Virgen del Rocío University Hospital, Seville, Spain
Wanderley Marques Bernardo, University of Sao Paulo, Sao Paulo, Brazil; Brazilian Medical Association, Sao Paulo, Brazil

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

4 Views

22 Dec 2022 | for Version 2

Ignacio Marín León, Internal Medicine, Virgen del Rocío University Hospital, Seville, Spain

4 Views Cite this report Responses(0)

Approved

I agree with the current version of the article

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Clinical Epidemiology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

12 Views

14 Dec 2022 | for Version 2

12 Views Cite this report Responses(0)

Approved

Are the rationale for, and objectives of, the Systematic Review clearly stated?

Yes
Are sufficient details of the methods and analysis provided to allow replication by others?

Yes
Is the statistical analysis and its interpretation appropriate?

Not applicable
Are the conclusions drawn adequately supported by the results presented in the review?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Evidence Based Medicine, Systematic Review and Guidelines

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

38 Views

04 Nov 2022 | for Version 1

Ignacio Marín León, Internal Medicine, Virgen del Rocío University Hospital, Seville, Spain

38 Views Cite this report Responses(1)

Approved With Reservations

The structure of this section does not agree with the established "norms" on how the "discussion" section should be drafted. The main findings of the study should first be highlighted in relation to the objective of the study, and in a second step explain the causes of such findings and contrast with similar studies in the available literature. Thus, the large first paragraph could be moved down (and shortened) in the text.
The reviewer does not agree with the statements made in the current third paragraph, in the sense of contrasting the effect on the "process of care" with the effect on "patient outcomes". There are RCTs that relate the implementation of guidelines process of care with the appropriateness and necessity of care for patients' benefits (i.e. see Romero et al. (2005)¹).
Some comment on study weaknesses is missing.

Are the rationale for, and objectives of, the Systematic Review clearly stated?

Yes
Are sufficient details of the methods and analysis provided to allow replication by others?

Partly
Is the statistical analysis and its interpretation appropriate?

Yes
Are the conclusions drawn adequately supported by the results presented in the review?

Yes

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Clinical Epidemiology

Respond to this report

Responses (1)

Author Response

19 Dec 2022

Anggie Ramirez, Universitat Autònoma de Barcelona, Barcelona, 08041, Spain

Dear Dr. Ignacio Marin,

We would like to thank you for agreeing to participate in the review of our research. We consider that your contributions are very valuable to improve our work.

Regarding your comments, we describe as follows the research team's position on each of them:

A. Concerning the analysis of the certainty of the evidence, you wrote:
Methods:
Data extraction, page 4, last paragraph, "An analysis of the quality of the evidence…we assessed the certainty...": It is unclear how this was done. GRADE is a very labour-intensive and specific methodology, and to perform a GRADE evaluation, the authors have two options: a) simply seek the strength of evidence from the original implemented guideline included in the review, or b) evaluate and critically assess original articles cited in the guidelines reviewed, when they did not use GRADE methodology. This needs clarification on what the authors mean. In addition, readers do not need the authors to explain what GRADE method is.
Results:
Quality of evidence assessment, page 7: This section should be clarified in accordance with previous comments on the use of GRADE.

Answer to #A:
We agree with you about the options using the GRADE method to develop or adapt a Clinical Practice Guideline. However, we used the GRADE method to assess the certainty of the evidence only for the RCTs that met the inclusion criteria. Therefore, it does not comprise part of the objective and scope of this systematic review to evaluate the evidence included in the CPGs that these studies reviewed.

We wrote as part of the Methods section:
The studies showing the lowest risk of bias (Irene et al., 2019) and one resulting in the highest risk of bias (Park et al., 2019) were chosen to be evaluated with the GRADE methodology and to build a summary of the findings. We decided to grade only four outcomes per study described, including two reporting statistically significant results and two not statistically significant results. Performing a GRADE table provided the rank of possible grades for the certainty of the evidence found in the 362 interventions from the 11 studies included.

Hence, we do not consider it necessary to modify the text in the section "Quality of evidence assessment". Then, we want to clarify that it is part of the objective and scope of this systematic review to evaluate the certainty of the evidence of the RCTs included and not of the EB-CPGs that they evaluated.

B. About the article's Discussion section, you wrote:
B.1
The structure of this section does not agree with the established "norms" on how the "discussion" section should be drafted. The main findings of the study should first be highlighted in relation to the objective of the study, and in a second step explain the causes of such findings and contrast with similar studies in the available literature. Thus, the large first paragraph could be moved down (and shortened) in the text.

Answer to #B.1:
It is relevant to clarify that our article complies with F1000Research guidelines for preparing a Systematic Review article. The journal requests the main body section as follows:
Main Body: The format of the main body of the article is flexible: it should be concise, making it easy to read and review, and presented in a format that is appropriate for the type of study presented. For most Systematic Reviews, the following standard format will be the most appropriate: Introduction, Methods, Results, Conclusions/Discussion.

Also, we consider that the aim of the discussion was achieved because we highlighted our findings and contrasted them with similar research. We must add that we have also complied with the PRISMA 2020 statement checklist (Page et al., 2021[1]). About the discussion, our study meets with the following items:
23a Provide a general interpretation of the results in the context of other evidence.
23b Discuss any limitations of the evidence included in the review.
23c Discuss any limitations of the review processes used.
23d Discuss the implications of the results for practice, policy, and future research.

Therefore, considering that neither of the indications describes an expected order of ideas as you have indicated, the authors alleged it convenient to include this first paragraph to provide a context about the world situation regarding the efforts to develop guidelines and the expectations placed by health caregivers on their effect. Also, to link this finding with the findings in our study.

Consequently, as the first paragraph of this section is congruent with the aim of this systematic review, we consider it appropriate to keep the paragraph unchanged.

B.2 Additionally, regarding your comment about the third paragraph of the discussion section:
The reviewer does not agree with the statements made in the current third paragraph, in the sense of contrasting the effect on the "process of care" with the effect on "patient outcomes". There are RCTs that relate the implementation of guidelines process of care with the appropriateness and necessity of care for patients' benefits (i.e. see Romero et al. (2005)[2]).

We would like to explain the following:
Answer #B.2:
This statement refers to the findings we got from the first systematic review (Ramírez-Morera et al., 2019[3]) that focused on the impact of CPGs in the cardiovascular approach; at that time, we included nine RCTs. Therefore, we decided to compare the results obtained in this second SR to our previous research regarding the number of interventions evaluated according to the dimensions described by Donabedian (1988)[4].

This report corresponds to the result of a Systematic Review that includes only Randomized Controlled Clinical Trials (RCTs) whose objective was to measure the impact that Clinical Practice Guidelines have produced on the quality of health care. This review's authors analyzed the RCT measurements' results according to the framework proposed by Avedis Donabedian^{^[5]} (7 January 1919 – 9 November 2000). Therefore, what has been understood by the implementation of clinical practice guidelines, as defined in the UK glossary published by NICE, adheres to the definition mentioned earlier: "implementation, The process of putting guideline recommendations into practice" (NICE 2014)^{^[6]}. Evaluation components of CPG use, such as clinical auditing, can be of great value. However, clinical auditing has not been disseminated to be available in other countries, predominantly low-income countries.
In 2008, Dutch authors Lugtenberg M, Burgers JS and Westert GP published an excellent systematic review^{^[7]} using the quality of care model by Donabedian. This publication has been very inspiring for the authors of this series of systematic reviews. This review mainly includes RCT studies and some before and after studies done on CPG made in the Netherlands, a country with a long tradition in the design and use of CPG. Some of their results are encouraging, although other aspects were weak, such as the impact on patient's health, which should be considered a very relevant result.

Due to the nature of the data, it was impossible to have a quantitative synthesis, only qualitative. But it did not inhibit us from assessing the certainty of the evidence for the results of the included studies after searching and screening according to the inclusion criteria.
To determine the confidence of the evidence, we have followed Cochrane's recommendations using the tools developed by GRADE. GRADE is a tool that requires knowledge of the essential elements of effective measurement and its potential biases. The authors of this SR are trained to complete this task with adequate competency.

B.3 You also wrote: Some comment on study weaknesses is missing.
Answer #B.3:
We did not write within a specific section of our article what the research's weaknesses are. The weaknesses of the study are described throughout different statements, for example, when describing the risk of bias assessment for the included studies. We also identified many EB-CPGs developed in contrast to the few numbers evaluated through formal RCT-type investigations.

We also wrote about the implications for research:
Due to recent research in this field, and the results of this study were not conclusive, more research is necessary to evaluate how EB-CPGs could impact the quality of health care, especially emphasising fewer investigated areas, such as the structure of healthcare services and patient outcomes.
Additionally, the design and methodological rigour applied to future RCTs should improve by considering the certainty of the evidence supporting the EB-CPGs recommendations. Besides, focusing on those with a greater level of evidence (high or moderate) which could lead to determining more clearly the effect that these recommendations have on the quality of health care.

C. About your minor comments:
C.1 First, about the abstract section you wrote:
Abstract: It is required that the clinical problem addressed in the study (breast cancer) appears in the abstract and keywords.

Answer #C.1:
We agree to add "breast cancer" as a keyword.

C.2 Second, concerning the introduction section, you wrote:
Introduction:
In the sixth paragraph: "Although survival has improved…..": There is a huge debate whether the improved survival depends on the early diagnosis or the improvement of knowledge of the natural history and new treatments of breast cancer (Gøtzsche et al. (2013)[8], Elmore et al. (2005)[9]). Thus, some sentences should be complemented with "...of early detection programs and treatment improvement".

Answer #C.2:
We agree to add the suggested text.

C.3 Third, referring to the discussion section, you wrote:
The comments on the usefulness of EB-CPG could be shortened.

Answer #C.3:
The authors alleged it is convenient to describe the usefulness of EB-CPG because it is congruent with the aim of this systematic review. Therefore, we disagree with the modification suggested.

C.4 Fourth and last, you wrote that there are 'partly' details of the methods and analysis provided to allow replication by others.

Answer #C.4:
Full details on methods and further analyses to allow replication by others can also be accessed through the research protocol, registered at PROSPERO: CRD42020205594.

References

[1] Page et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical Research Ed.), 372, p.n71. [Online]. Available at: doi:10.1136/bmj.n71.

[2] Romero A, Alonso C, Marín I, Grimshaw J, et al.: Effectiveness of a Multifactorial Strategy for Implementing Clinical Guidelines on Unstable Angina: Cluster Randomized Trial. Revista Española de Cardiología (English Edition). 2005; 58 (6): 640-648

[3] Ramírez-Morera, A., Tristan, M. and Vazquez, J. C. (2019). Effects of evidence-based clinical practice guidelines in cardiovascular health care quality improvements: A systematic review [version 3; peer review: 2 approved]. F1000Research 2019, 8:1041[Online]. Available at: https://doi.org/10.12688/f1000research.18865.3

[4] Donabedian, A. (1988). The quality of care. How can it be assessed? The Journal of the American Medical Association, 260 (12), pp.1743–1748. [Online]. Available at: doi:10.1001/jama.260.12.1743.

[5] Donabedian model. (2022, August 8). In Wikipedia. https://en.wikipedia.org/wiki/Donabedian_model

[6] Glossary | Developing NICE guidelines: the manual | Guidance | NICE [WWW Document], 2014 LAST UPDATED 18 January 2022. URL https://www.nice.org.uk/process/pmg20/chapter/glossary

[7] Lugtenberg, M., Burgers, J.S., Westert, G.P., 2009. Effects of evidence-based clinical practice guidelines on quality of care: a systematic review. Qual. Saf. Health Care 18, 385–392. doi:10.1136/qshc.2008.028043

[8] Gøtzsche PC, Jørgensen KJ: Screening for breast cancer with mammography. Cochrane Database Syst Rev. 2013. CD001877

[9] Elmore JG, Armstrong K, Lehman CD, Fletcher SW: Screening for breast cancer. JAMA. 2005; 293 (10): 1245-56

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] Alonso-Coello P, et al.: The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual. Saf. Health Care. 2010; 19(6): e58. PubMed Abstract | Publisher Full Text

[2] Boekhout AH, et al.: A survivorship care plan for breast cancer survivors: extended results of a randomised clinical trial. J. Cancer Surviv. 2015; 9(4): 683–691. PubMed Abstract | Publisher Full Text

[3] Donabedian A: The quality of care. How can it be assessed? J. Am. Med. Assoc. 1988; 260(12): 1743–1748. Publisher Full Text

[4] Cochrane Effective Practice and Organisation of Care (EPOC): Data Collection Checklist. EPOC Resources for review authors, 2017. (updated March 2019). Cochrane.Reference Source

[5] Greenlee H, et al.: Survivorship care plans and adherence to lifestyle recommendations among breast cancer survivors. J. Cancer Surviv. 2016; 10(6): 956–963. PubMed Abstract | Publisher Full Text

[6] Ghoncheh M, Pournamdar Z, Salehiniya H: Incidence and mortality and epidemiology of breast cancer in the world. Asian Pac. J. Cancer Prev. 2016; 17(S3): 43–46. Publisher Full Text

[7] GRADEpro GDT: GRADEpro Guideline Development Tool: [Computer program]. Hamilton (ON): McMaster University (developed by Evidence Prime, Inc.).2021.Reference Source

[8] Grimshaw JM, Russell IT: Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993; 342(8883): 1317–1322. PubMed Abstract | Publisher Full Text

[9] Grunfeld E, et al.: Clinical practice guidelines for the care and treatment of breast cancer: follow-up after treatment for breast cancer (summary of the 2005 update). CMAJ. 2005; 172(10): 1319–1320. PubMed Abstract | Publisher Full Text

[10] Grunfeld E, et al.: Evaluating survivorship care plans: results of a randomised, clinical trial of patients with breast cancer. J. Clin. Oncol. 2011; 29(36): 4755–4762. PubMed Abstract | Publisher Full Text

[11] Hershman DL, et al.: Randomised controlled trial of a clinic-based survivorship intervention following adjuvant therapy in breast cancer survivors. Breast Cancer Res. Treat. 2013; 138(3): 795–806. PubMed Abstract | Publisher Full Text

[12] Higgins JPT, et al.: Cochrane Handbook for Systematic Reviews of Interventions version 6.3. (updated February 2022). Cochrane.2022a.Reference Source

[13] Higgins JPT, et al.:Chapter 6: Choosing effect measures and computing estimates of effect.Higgins JPT, Thomas J, Chandler J, et al. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane;2022b.Reference Source

[14] Higgins JPT, et al.:Chapter 8: Assessing risk of bias in a randomised trial.Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane;2022c.Reference Source

[15] Higgins JPT, et al.:Chapter 23: Including variants on randomised trials.Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane;2022d.Reference Source

[16] Ivers N, et al.: Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst. Rev. 2012; 6: CD000259. PubMed Abstract | Publisher Full Text

[17] Institute of Medicine (US) Committee to Advise the Public Health Service on Clinical Practice Guidelines: Clinical practice guidelines: directions for a new program. Field MJ, Lohr KN, editors.Washington (DC):National Academies Press (US);1990. Publisher Full Text

[18] Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines: Clinical practice guidelines we can trust. Graham R, et al., editors.Washington (DC):National Academies Press (US);2011. Publisher Full Text

[19] Irene S, H., et al.: Efficacy of a web-based women’s health survivorship care plan for young breast cancer survivors: a randomised controlled trial. Breast Cancer Res. Treat. 2019; 176(3): 579–589. PubMed Abstract | Publisher Full Text

[20] Khatcheressian JL, et al.: American Society of Clinical Oncology 2006 update of the breast cancer follow-up and management guidelines in the adjuvant setting. J. Clin. Oncol. 2006; 24(31): 5091–5097. PubMed Abstract | Publisher Full Text

[21] Khatcheressian JL, et al.: Breast cancer follow-up and management after primary treatment: American Society of Clinical Oncology clinical practice guideline update. J. Clin. Oncol. 2013; 31(7): 961–965. PubMed Abstract | Publisher Full Text

[22] Klinkhammer-Schalke M, et al.: Direct improvement of quality of life using a tailored quality of life diagnosis and therapy pathway: randomised trial in 200 women with breast cancer. Br. J. Cancer. 2012; 106(5): 826–838. PubMed Abstract | Publisher Full Text

[23] Knobf MT, et al.: Effect of a randomised controlled exercise trial on bone outcomes: influence of adjuvant endocrine therapy. Breast Cancer Res. Treat. 2016; 155(3): 491–500. PubMed Abstract | Publisher Full Text

[24] Kwan TJ: Do evidence-based guidelines bridge the gaps in research?. Bridging the gaps. Abstracts of the 12th Cochrane Colloquium. 2004 2-6 Oct; Ottawa, Canada. 2004.

[25] Loren AW, et al.: Fertility preservation for patients with cancer: American Society of Clinical Oncology clinical practice guideline update. J. Clin. Oncol. 2013; 31(19): 2500–2510. PubMed Abstract | Publisher Full Text

[26] Linskey ME: Defining excellence in evidence-based medicine clinical practice guidelines. Clin. Neurosurg. 2010; 57: 28–37. PubMed Abstract

[27] Lugtenberg M, Burgers JS, Westert GP: Effects of evidence-based clinical practice guidelines on quality of care: a systematic review. Qual. Saf. Health Care. 2009; 18(5): 385–392. Publisher Full Text

[28] Maly RC, et al.: Randomized Controlled Trial of Survivorship Care Plans Among Low-Income, Predominantly Latina Breast Cancer Survivors. J. Clin. Oncol. 2017; 35(16): 1814–1821. PubMed Abstract | Publisher Full Text

[29] O’Brien MA, et al.: Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst. Rev. 2007; 4: CD000409. PubMed Abstract | Publisher Full Text

[30] Page, et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical Research Ed.). 2021; 372: n71. PubMed Abstract | Publisher Full Text

[31] Park J-H, et al.: The effect of oncologists’ exercise recommendations on the level of exercise and quality of life in survivors of breast and colorectal cancer: A randomised controlled trial. Cancer. 2015; 121(16): 2740–2748. PubMed Abstract | Publisher Full Text

[32] Park S-H, et al.: Adherence to American cancer society guidelines on nutrition and physical activity in female cancer survivors: results from a randomised controlled trial (yale fitness intervention trial). Cancer Nurs. 2019; 42(3): 242–250. PubMed Abstract | Publisher Full Text

[33] Ramírez-Morera A, Tristan M, Vazquez JC: Effects of evidence-based clinical practice guidelines in cardiovascular health care quality improvements: A systematic review [version 3; peer review: 2 approved]. F1000Res. 2019; 8: 1041. PubMed Abstract | Publisher Full Text

[34] Ramírez-Morera A, et al.: Extended data for the second SR CPG Breast Cancer. osf.io/6h9pm.2022.

[35] Review Manager (RevMan): [Computer program]. Version 5.4.1 for Mac. The Cochrane Collaboration;2020.

[36] Ricci-Cabello I, et al.: Adherence to breast cancer guidelines is associated with better survival outcomes: a systematic review and meta-analysis of observational studies in EU countries. BMC Health Serv. Res. 2020; 20(1): 920. PubMed Abstract | Publisher Full Text

[37] Schünemann H, et al.: GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group. 2013.2013.Reference Source

[38] Schmitz KH, et al.: American College of Sports Medicine roundtable on exercise guidelines for cancer survivors. Med. Sci. Sports Exerc. 2010; 42(7): 1409–1426. PubMed Abstract | Publisher Full Text

[39] Sciwheel Reference Manager & Generator: Computer program. MacOS Monterey. London:Sciwheel Ltd.;2022.

[40] Smith-Turchyn J, et al.: Bridging the gap: incorporating exercise evidence into clinical practice in breast cancer care. Support Care Cancer. 2020; 28(2): 897–905. PubMed Abstract | Publisher Full Text

[41] Sterne JAC, et al.: RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ (Clinical Research Ed.). 2019; 366: l4898. PubMed Abstract | Publisher Full Text

[42] Stark SS, et al.: Randomised controlled trial of the effect of a reproductive health survivorship care plan on fertility and pregnancy concerns, vasomotor symptoms, sexual health, and contraception in young breast cancer survivors. Fertil. Steril. 2018; 110(4): e48. Publisher Full Text

[43] Watters W: Defining evidence-based clinical practice guidelines. “AOOS Now”.2008. American Academy of Orthopaedic Surgeons. RESEARCH2.Reference Source Reference Source

[44] Weisz G, et al.: The Emergence of Clinical Practice Guidelines. Milbank Q. 2007; 85(4): 691–727. PubMed Abstract | Publisher Full Text

[45] WHO: Breast Cancer. Cancer prevention. Early Diagnosis and Screening. Geneva:World Health Organization;2020.Reference Source

[46] Woolf SH, et al.: Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ (Clinical Research Ed.). 1999; 318(7182): 527–530. PubMed Abstract | Publisher Full Text | Free Full Text

[47] Worrall G, Chaulk P, Freake D: The effects of clinical practice guidelines on patient outcomes in primary care: a systematic review. Can. Med. Assoc. J. 1997; 156(12): 1705–1712. PubMed Abstract

Effects of evidence-based clinical practice guidelines for breast cancer in health care quality improvements. A second systematic review.

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Methods

Study search

Table 1. Structure of the clinical question.

Studies selection

Data extraction

Results

Study identification and selection

Figure 1. PRISMA flowchart of the studies selection process (PRISMA; Page et al., 2021).

Characteristics of the included studies

Table 2. Characteristics of all included studies (n=11).

Figure 2. Analysis of the Risk of Bias (RoB2) for all included studies (n=11).

Quality of evidence assessment

Table 3a. Summary of findings: Low risk of bias (Irene et al., 2019).

Table 3b. Summary of findings: High risk of bias (Park et al., 2019).

Assessment of the studies outcomes

Table 4. Summary of results for all included studies by dimensions and effect (n=11).

Discussion

Conclusions

Implications for the practice

Implications for the research

Data (and software) availability

Data availability

Author contributions

Leading author information

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated