Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol

Raju Kanukula; Matthew J. Page; Kerry Dwan; Simon L. Turner; Elizabeth Loder; Evan Mayo-Wilson; Tianjing Li; Adya Misra; Steve McDonald; Andrew B. Forbes; Joanne E. McKenzie

doi:10.12688/f1000research.53034.1

Home Browse Development of a checklist to detect errors in meta-analyses in systematic...

ALL Metrics

-

Views

-

Downloads

Get PDF

Get XML

Export

▬

✚

Study Protocol

Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol

[version 1; peer review: 2 approved]

Raju Kanukula¹, Matthew J. Page¹, Kerry Dwan², [...] Simon L. Turner¹, Elizabeth Loder^3,4, Evan Mayo-Wilson⁵, Tianjing Li^6,7, Adya Misra⁸, Steve McDonald¹, Andrew B. Forbes¹, Joanne E. McKenzie ¹

Raju Kanukula¹, Matthew J. Page¹, [...] Kerry Dwan², Simon L. Turner¹, Elizabeth Loder^3,4, Evan Mayo-Wilson⁵, Tianjing Li^6,7, Adya Misra⁸, Steve McDonald¹, Andrew B. Forbes¹, Joanne E. McKenzie ¹

PUBLISHED 08 Jun 2021

Author details Author details

¹ School of Public Health and Preventive Medicine, Monash University, Melbourne, VICTORIA, 3004, Australia
² Cochrane Methods Support Unit, Cochrane, London, SW1Y 4QX, UK
³ Division of Headache, Department of Neurology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, 02115, USA
⁴ The BMJ, BMA House, London, WC1H 9JP, UK
⁵ Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, 47405, USA
⁶ Department of Ophthalmology, School of Medicine, University of Colorado Denver, Denver, CO, 80045, USA
⁷ Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, 21205, USA
⁸ PeerJ, Inc, San Diego, CA, 92191, USA

Raju Kanukula
Roles: Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Matthew J. Page
Roles: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Kerry Dwan
Roles: Writing – Review & Editing

Simon L. Turner
Roles: Writing – Review & Editing

Elizabeth Loder
Roles: Writing – Review & Editing

Evan Mayo-Wilson
Roles: Writing – Review & Editing

Tianjing Li
Roles: Writing – Review & Editing

Adya Misra
Roles: Writing – Review & Editing

Steve McDonald
Roles: Writing – Review & Editing

Andrew B. Forbes
Roles: Writing – Review & Editing

Joanne E. McKenzie
Roles: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

Abstract

Background: Systematic reviews underpin clinical practice and policies that guide healthcare decisions. A core component of many systematic reviews is meta-analysis, which is a statistical synthesis of results across studies. Errors in the conduct and interpretation of meta-analysis can lead to incorrect conclusions regarding the benefits and harms of interventions; and studies have shown that these errors are common. Enabling peer reviewers to better detect errors in meta-analysis through the use of a checklist provides an opportunity for these errors to be rectified before publication. To our knowledge, no such checklist exists.
Objective: To develop and evaluate a checklist to detect errors in pairwise meta-analyses in systematic reviews of interventions.
Methods: We will undertake a four-step process to develop the checklist. First, we will undertake a systematic review of studies that have evaluated errors in the conduct and interpretation of meta-analysis to generate a bank of items to consider for the checklist. Second, we will undertake a survey of systematic review methodologists and statisticians to seek their views on which items, of the bank of items generated in step 1, are most important to include in the checklist. Third, we will hold a virtual meeting to agree upon which items to include in the checklist. Fourth, before finalising the checklist, we will pilot with editors and peer reviewers of journals.
Conclusion: The developed checklist is intended to help journal editors and peer reviewers identify errors in the application and interpretation of meta-analyses in systematic reviews. Fewer errors in the conduct and improved interpretation will lead to more accurate review findings and conclusions to inform clinical practice.

Keywords

systematic review, meta-analysis, errors, checklist, reporting guideline, statistical issues, synthesis, pair-wise meta-analysis

Corresponding author: Joanne E. McKenzie

Competing interests: No competing interests were disclosed.

Grant information: The author(s) declared that no grants were involved in supporting this work.

Copyright: © 2021 Kanukula R et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Kanukula R, Page MJ, Dwan K et al. Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol [version 1; peer review: 2 approved]. F1000Research 2021, 10:455 (https://doi.org/10.12688/f1000research.53034.1) First published: 08 Jun 2021, 10:455 (https://doi.org/10.12688/f1000research.53034.1) Latest published: 08 Jun 2021, 10:455 (https://doi.org/10.12688/f1000research.53034.1)

1. Introduction

Systematic reviews (SRs) frequently underpin clinical practice guidelines and policies that guide healthcare decisions. A core component of many SRs is meta-analysis, a statistical technique used to synthesise study effect estimates from studies addressing similar questions, yielding a quantitative summary.¹ Extensions to meta-analysis (e.g. meta-regression, subgroup analysis) allow for investigation of factors that may explain variation of results across studies. These methods have the potential to provide valuable insights for healthcare decision-making; however, they are reliant on the methods being appropriately applied and interpreted.

Many errors can arise when conducting meta-analysis. For example, when meta-analysing continuous outcomes, calculations may be incorrect if standard errors are confused with standard deviations. When data are included from multi-arm trials, there is the risk that participants might be counted more than once when multiple comparisons from these trials are eligible for inclusion in the same meta-analysis. For example, from a three-arm trial of paroxetine, fluoxetine, and placebo, two comparisons would be eligible for a meta-analysis of ‘antidepressants versus placebo’ (i.e., ‘paroxetine versus placebo’ and ‘fluoxetine versus placebo’). When dealing with non-standard randomized trials – such as crossover trials, cluster-randomized trials, or split-body trials – there is a risk that variances of the effect estimates in the meta-analysis do not appropriately account for the correlation in observations induced by these designs.²^–⁵ Such errors can lead to studies receiving the incorrect weight in the meta-analysis with potential consequent impact on the combined estimate of intervention effect and its confidence interval, and other statistics, such as the estimated heterogeneity variance and measures of inconsistency. In some circumstances, these errors will lead to a different interpretation of the findings and review conclusions.⁶

Statistical errors have been observed frequently in published SRs. For example, a study including 42 reviews from the Cochrane Cystic Fibrosis and Genetic Disorders Group found that nearly half of the SRs had at least one error (e.g., used standard error instead of standard deviation; calculated standard deviations incorrectly from the standard error given in the report; entered median instead of mean).⁷ Another study in which the authors re-extracted the data from two randomly selected trials included in each of 27 meta-analyses, found errors in how the meta-analyst entered data for at least one of the two trials in 17 (63%) of the meta-analyses.⁶ Furthermore, some published meta-analyses papers have been retracted because of errors in analyses and error in results and/or conclusions.⁸^–¹⁰

Research has shown that errors in the interpretation of statistical analyses in reviews are also common. For example, of 110 SRs indexed in MEDLINE® in 2014, 62 used the random-effects model, but 57 (92%) incorrectly interpreted the meta-analytic effect as the best estimate of a common intervention effect across studies, rather than as the average of the intervention effects across studies. In 42 of the 110 meta-analyses, a subgroup analysis was undertaken, but the findings were not interpreted with respect to a test for interaction in 29/42 (69%), and in 11/42 (26%), a claim of a subgroup difference was made based on a statistically significant effect in one group and not the other.¹¹ Furthermore, the potential for confounding by other factors as a possible explanation for observed subgroup effects, was not raised in any of the SRs.¹¹^,¹²

Many strategies have been proposed to improve the conduct of meta-analysis (thus lessening the chance of errors occurring) and the interpretation of findings. These include, for example, textbooks,¹³^–¹⁵ training on meta-analysis methods, connection with support systems (e.g., Cochrane’s TaskExchange), and the inclusion of statisticians on review teams. However, even with these strategies, errors will still occur. A possible additional strategy is to enable peer reviewers to better detect possible errors in meta-analyses.

The peer-review process is regarded as a valuable approach for helping peer reviewers and journal editors to judge the quality, critically appraise and finally accept or reject the submitted manuscripts for publication.¹⁶ Researchers have explored the impact of checklists to guide peer reviewers in assessing the completeness of reporting of submitted manuscripts, and have found some evidence that these are effective.¹⁷^–¹⁹ For example, training early career researchers to use the COBPeer tool (which is an online CONSORT-based peer-review tool assessing nine domains: the eight most important CONSORT domains and a switch in primary outcome(s)) helped them detect inadequate reporting in randomized trials compared to the usual review process.¹⁷^,²⁰ To our knowledge, no such checklist has been developed to detect statistical errors in meta-analyses.

Aim: To develop and evaluate a checklist to detect conduct and interpretation errors in pairwise meta-analyses in systematic reviews of interventions.

2. Defining the concept of statistical errors

The notion of statistical conduct and interpretation errors is not simple. Brown and colleagues²¹ defined errors to be “actions or conclusions that are demonstrably and unequivocally incorrect from a logical or epistemological point of view (e.g. … mathematical mistakes, statements not supported by the data, incorrect statistical procedures …)”. In this research, we will consider statistical errors to include those arising from underlying assumptions not being met, incorrect values used in the calculations, application of incorrect statistical methods, and misinterpretation of the results and statistical tests. We plan to initially group errors into categories (Section 3.2.4) and refine and revise these based on the types of errors identified through the systematic review (Section 3.2). Our focus will be on errors where it can be reasonably expected that a trained meta-analyst should have or could have known better, recognising that there is subjectivity in making this determination.²¹

3. Methods

3.1 Contributors

A core team (RK, MJP, KD, SLT, EL, EMW, TL, AM, ABF, JEM) will lead the development of this checklist. The core team will conduct the systematic review, develop survey content and analyse survey responses, draft the checklist, coordinate piloting of the checklist, and decide the final content of the checklist. The core team consists of individuals with experience in meta-analysis methods and SR methodology, contributors of the Cochrane Handbook for Systematic Reviews of Interventions, and editors of medical journals who frequently publish SRs (BMJ, PLOS Medicine, Cochrane Database of Systematic Reviews, American Journal of Public Health, and Systematic Reviews).

3.2 Systematic review

We will conduct an SR of studies evaluating errors in the conduct and interpretation of pairwise meta-analysis, for the purpose of identifying types of errors, their prevalence, and to generate a bank of items to potentially be included in the checklist.

3.2.1 Eligibility criteria

Studies will need to meet the following eligibility criteria:

Inclusion criteria:

• Studies evaluating types of errors (and potentially their prevalence) in the conduct and interpretation of meta-analyses (and its extensions, for example, subgroup analysis, sensitivity analysis) in SRs of interventions (irrespective of included study design);
• Articles presenting a checklist or tool to evaluate the conduct of meta-analyses in SRs.

Exclusion criteria:

• Studies evaluating the methodological or reporting quality or risk of bias in SRs using a tool that does not specifically examine statistical errors (e.g. PRISMA, AMSTAR-2, ROBIS);
• Studies or checklists evaluating errors in statistical analyses in primary study designs (e.g. randomized trials and observational studies).

3.2.2 Search methods

We will search MEDLINE, Embase and Scopus from inception to January 2021, without any publication type or language restrictions. The search strategies for MEDLINE and Embase combine subject heading terms and text words related to statistical errors in meta-analyses are presented in Appendix (see Extended data).²² The search was iteratively developed and tested by an experienced information specialist (SM) using a set of 10 methods articles relevant to the topic. The Scopus search closely replicates the MEDLINE search with respect to included terms and word adjacency but is limited to the following subject areas in Scopus: medicine, nursing, dentistry and health professions.

We will also search abstracts of papers and posters presented at Cochrane Colloquia since 2011 (available at https://abstracts.cochrane.org/), including the Global Evidence Summit 2017. The search strategy will be determined by assessing the relative recall of terms from eligible studies identified from searches of MEDLINE, Embase and Scopus. We will screen the reference lists and conduct a cited reference search in Web of Science of included articles and review our personal collections of reports or studies related to statistical issues in meta-analyses. In addition, we will contact organisations that produce SRs (e.g., Cochrane, Campbell Collaboration, National Institute for Health and Care Excellence), and journals that frequently publish SRs to seek any in-house checklists they are willing to share.

3.2.3 Selection of studies

Two authors will screen independently all titles and abstracts according to the aforementioned eligibility criteria and retrieve the full text of any potentially relevant articles. The same authors will screen the full texts of retrieved articles. In case of any discrepancies, a senior author will adjudicate and finalise the inclusion or exclusion of any article(s).

3.2.4 Data collection

Once we finalise the studies to be included, two authors will collect data independently from each article using a standardised data collection form. For studies evaluating types of errors (and potentially their prevalence), we will collect the following information: corresponding author name, email address, year of publication, journal name, objective(s), focus of error investigation (e.g., multi-arm trials, cross-over trials, cluster randomized trials), type and prevalence of errors, and recommendations provided for conducting meta-analyses. For articles presenting a checklist or tool to evaluate conduct or reporting of meta-analyses in SRs, we will collect the following information: checklist/tool name, method of checklist/tool development, number of items included in the checklist/tool and scope of the checklist/tool. In addition, we will collect all the items and response options pertinent to meta-analysis and its interpretation identified in the tools/checklists, and these will be added to our item bank.

Once we have extracted data from all articles, we will review the items in the item bank and remove any duplicate or redundant items. We will then group items into broader categories. For example, those relating to data type (e.g., continuous, binary), rare outcomes (i.e., handling of zero events in one or both arms), design of included studies (e.g. cross-over, cluster, multi-arm, non-randomised [e.g. interrupted time series, cohort]), type of analysis (meta-analysis, subgroup analysis, meta-regression, sensitivity analysis, publication bias analysis) and issues of interpretation. We will use Microsoft Excel 2016 for data management.

3.3 Survey

We will send an invitation to all SR methodologists and statisticians (identified from the Cochrane Methods community, Campbell Methods Coordinating Group, Society for Research Synthesis Methodology and other SR methodologists and statisticians known to the core team members) and SR editors and statistical editors (identified from Cochrane Review Groups, and those supporting other journals that frequently publish SRs) to complete a survey to inform the development of the checklist. The survey will ask respondents to:

1) Provide their views on the most important items from the bank of items (generated from the systematic review) to include in the checklist. We will ask respondents to prioritise items that capture/identify the most common and consequential errors expected to occur in the conduct and interpretation of meta-analyses;
2) Provide their views on specific signals (or ‘red flags’) that might lead them to conduct a more thorough investigation of statistical errors in reviews (e.g., size of the effect for some studies, meta-analysis methods used, I-squared value etc.).

We will ask researchers to provide rationale for their responses and to suggest additional items not listed in the survey. We will calculate frequencies of each response option for each item and specific signals. For an item or specific signal to meet consensus for discussion, one of the response options for the item or signal will need to be selected by more than 70% of survey respondents; this threshold was selected according to Sumsion 1998 et al.²³

3.4 Virtual meeting

Following the survey, the core team will hold a virtual meeting to agree upon which items to include in the statistical errors’ checklist for editors/peer reviewers, which items might trigger further investigation (by a statistical reviewer or the authors) and discuss how best to word each item. Attendees will discuss all items exceeding the 70% threshold in the survey. We will also send the items rated as important by fewer than 70% to meeting attendees prior to the meeting, to provide them with the chance of “rescuing” items for discussion at the meeting. Following the meeting, the core team will draft the checklist and an accompanying guidance document (with examples for each item).

3.5 Piloting

In the first stage of piloting, two reviewers will read the draft checklist and guidance document before independently applying it to a random sample of 20 reviews; 10 from each of two previous methodological studies that collated systematic reviews. Specifically, the first methodological study includes 42 SRs of nutrition research (the ROBUST study)²⁴ that were published between January 2018 and June 2019, and the second includes 31 SRs of interventions for arthritis, depression or anxiety (the SIM study) that were published between January 2010 and January 2012.²⁵ The reviewers will record issues on whether the wording of items is ambiguous or difficult to interpret, and those items will be discussed by the core team and improved.

In the second stage of piloting, we will invite associate editors and peer reviewers of journals that frequently handle SR submissions to pilot the checklist and provide feedback on its usability. After collating the feedback received from the peer reviewers and editors, we will finalise the checklist and the accompanying guidance document.

3.6 Dissemination and knowledge translation strategy

We plan to publish the developed checklist and guidance document in an open-access journal. We will disseminate the checklist via presentations and workshops at relevant conferences and workshops focused on SR methodology, health technology assessment and evidence-based medicine (e.g., Cochrane Colloquia, Evidence Live, HTAi Annual Meeting, The Global Evidence Summit, G-I-N Conference), and via social media, and in a series of international webinars. We also plan to create a user-friendly, online version of the checklist and guidance document for use by journals that publish SRs and meta-analyses.

The developed checklist and related work will be published in open access journals. Associated datasets, data collection forms and analyses not included in any publication will be made publicly available via an online repository.

4. Conclusion

The developed checklist could help journal editors and peer reviewers identify errors in the conduct and interpretation of meta-analyses in SRs. Fewer errors and improved interpretation would lead to more accurate SR findings and conclusions to inform clinical practice.

Study status: We have completed searching and screening for sections 3.2.1, 3.2.2 and 3.2.3 for systematic review.

Data availability

Underlying data

No underlying data are associated with this article.

Extended data

Bridges: Search strategy for systematic review of studies evaluating errors in the conduct and interpretation of pairwise meta-analysis. Monash University. DOI: https://doi.org/10.26180/14446293.v1.²²

This project contains the following extended data:

- Search strategy for systematic review of studies evaluating errors in the conduct and interpretation of pairwise meta-analysis

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

References

1. McKenzie JE, Brennan SE: Chapter 12: Synthesizing and presenting findings using other methods. Cochrane Handbook for Systematic Reviews of Interventions. 2019: 321–47.
2. Richardson M, Garner P, Donegan S: Cluster Randomised Trials in Cochrane Reviews: Evaluation of Methodological and Reporting Practice. PLoS One. 2016; 11(3): e0151818. PubMed Abstract | Publisher Full Text | Free Full Text
3. Donner A, Klar N: Issues in the meta-analysis of cluster randomized trials. Stat Med. 2002; 21(19): 2971–80. PubMed Abstract | Publisher Full Text
4. Stedman MR, Curtin F, Elbourne DR, et al.: Meta-analyses involving cross-over trials: methodological issues. Int J Epidemiol. 2011; 40(6): 1732–4. PubMed Abstract | Publisher Full Text
5. Nolan SJ, Hambleton I, Dwan K: The use and reporting of the cross-over study design in clinical trials and systematic reviews: a systematic assessment. PLoS One. 2016; 11(7): e0159014. PubMed Abstract | Publisher Full Text | Free Full Text
6. Hrobjartsson A, Gotzsche PC: Unsubstantiated claims of large effects of placebo on pain: serious errors in meta-analysis of placebo analgesia mechanism studies. J Clin Epidemiol. 2006; 59(4): 336–8. discussion 9–41. PubMed Abstract | Publisher Full Text
7. Jones AP, Remmington T, Williamson PR, et al.: High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol. 2005; 58(7): 741–2. PubMed Abstract | Publisher Full Text
8. Roba AA, Tefera M, Worku T, et al.: Retraction Note: Application of 4% chlorhexidine to the umbilical cord stump of newborn infants in lower income countries: a systematic review and meta-analysis. Matern Health Neonatol Perinatol. 2020; 6: 4. PubMed Abstract | Publisher Full Text | Free Full Text
9. Guo LQ, Chen Y, Mi BB, et al.: Retraction Note to: Ambient air pollution and adverse birth outcomes: a systematic review and meta-analysis. J Zhejiang Univ Sci B. 2020; 21(9): 756. PubMed Abstract | Publisher Full Text | Free Full Text
10. Ezenwa BN, Bello S, Oyenusi EE, et al.: RETRACTED: Galactogogues use Among Mothers With Preterm Births: A Systematic Review and Meta-Analysis. J Hum Lact. 2020; 36(3): NP3–NP14. PubMed Abstract | Publisher Full Text
11. Page MJ, Altman DG, McKenzie JE, et al.: Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common: a cross-sectional analysis. J Clin Epidemiol. 2018; 95: 7–18. PubMed Abstract | Publisher Full Text
12. Fisher DJ, Carpenter JR, Morris TP, et al.: Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach? BMJ. 2017; 356. PubMed Abstract | Publisher Full Text | Free Full Text
13. Cooper H, Hedges LV, Valentine JC: The handbook of research synthesis and meta-analysis: Russell Sage Foundation; 2019.
14. Schmid CH, White IR, Stijnen T: Introduction to Systematic Review and Meta-Analysis. Handbook of Meta-Analysis. 2020.
15. Higgins JP, Thomas J, Chandler J, et al.: Cochrane handbook for systematic reviews of interventions. John Wiley & Sons; 2019.
16. Rennie D: Editorial peer review in biomedical publication: the first international congress. JAMA. 1990; 263(10): 1317. Publisher Full Text
17. Chauvin A, Ravaud P, Moher D, et al.: Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study. BMC Medicine. 2019; 17(1): 205. PubMed Abstract | Publisher Full Text | Free Full Text
18. Speich B, Schroter S, Briel M, et al.: Impact of a short version of the CONSORT checklist for peer reviewers to improve the reporting of randomised controlled trials published in biomedical journals: study protocol for a randomised controlled trial. BMJ Open. 2020; 10(3): e035114. PubMed Abstract | Publisher Full Text | Free Full Text
19. Blanco D, Schroter S, Aldcroft A, et al.: Effect of an editorial intervention to improve the completeness of reporting of randomised trials: a randomised controlled trial. BMJ Open. 2020; 10(5): e036799. PubMed Abstract | Publisher Full Text | Free Full Text
20. Chauvin A, Moher D, Altman D, et al.: A protocol of a cross-sectional study evaluating an online tool for early career peer reviewers assessing reports of randomised controlled trials. BMJ Open. 2017; 7(9): e017462. PubMed Abstract | Publisher Full Text | Free Full Text
21. Brown AW, Kaiser KA, Allison DB: Issues with data and analyses: Errors, underlying themes, and potential solutions. Proc Natl Acad Sci U S A. 2018; 115(11): 2563–70. PubMed Abstract | Publisher Full Text | Free Full Text
22. Page M, Kanukula R, McKenzie J: Search strategy for systematic review of studies evaluating errors in the conduct and interpretation of pairwise meta-analysis. Monash University. Online resource. 2021. [database on the Internet]. Publisher Full Text
23. Sumsion T: The Delphi technique: an adaptive research tool. Br J Occup Ther. 1998; 61(4): 153–6. Publisher Full Text
24. Page MJ, Bero L, Kroeger CM, et al.: Investigation of Risk Of Bias due to Unreported and SelecTively included results in meta-analyses of nutrition research: the ROBUST study protocol. F1000Res. 2019; 8(1760): 1760. PubMed Abstract | Publisher Full Text | Free Full Text
25. Page MJ, Forbes A, Chau M, et al.: Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study. BMJ Open. 2016; 6(4): e011863. PubMed Abstract | Publisher Full Text | Free Full Text

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 08 Jun 2021

Author details Author details

¹ School of Public Health and Preventive Medicine, Monash University, Melbourne, VICTORIA, 3004, Australia
² Cochrane Methods Support Unit, Cochrane, London, SW1Y 4QX, UK
³ Division of Headache, Department of Neurology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, 02115, USA
⁴ The BMJ, BMA House, London, WC1H 9JP, UK
⁵ Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, 47405, USA
⁶ Department of Ophthalmology, School of Medicine, University of Colorado Denver, Denver, CO, 80045, USA
⁷ Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, 21205, USA
⁸ PeerJ, Inc, San Diego, CA, 92191, USA

Raju Kanukula
Roles: Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Matthew J. Page
Roles: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Kerry Dwan
Roles: Writing – Review & Editing

Simon L. Turner
Roles: Writing – Review & Editing

Elizabeth Loder
Roles: Writing – Review & Editing

Evan Mayo-Wilson
Roles: Writing – Review & Editing

Tianjing Li
Roles: Writing – Review & Editing

Adya Misra
Roles: Writing – Review & Editing

Steve McDonald
Roles: Writing – Review & Editing

Andrew B. Forbes
Roles: Writing – Review & Editing

Joanne E. McKenzie
Roles: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Article Versions (1)

version 1

Published: 08 Jun 2021, 10:455

https://doi.org/10.12688/f1000research.53034.1

Copyright

© 2021 Kanukula R et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

0

SEE MORE DETAILS

CITE

how to cite this article

Kanukula R, Page MJ, Dwan K et al. Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol [version 1; peer review: 2 approved]. F1000Research 2021, 10:455 (https://doi.org/10.12688/f1000research.53034.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 08 Jun 2021

Views

38

Reviewer Report 05 Jul 2021

Therese D. Pigott, School of Public Health and College of Education and Human Development, Georgia State University, Atlanta, GA, USA

Approved

https://doi.org/10.5256/f1000research.56375.r87112

This work will provide an important contribution to the field, highlighting typical errors that can occur in pairwise meta-analyses of intervention effects. One question I have is the audience of the checklist. The study protocol is situated in meta-analysis for ... Continue reading

This work will provide an important contribution to the field, highlighting typical errors that can occur in pairwise meta-analyses of intervention effects. One question I have is the audience of the checklist. The study protocol is situated in meta-analysis for health and medical interventions. The search strategy for relevant systematic reviews for stage one of the process focuses on the health and medical literature, with only the Campbell Collaboration as a source for meta-analyses in the social sciences. If the intent of the checklist is to apply more broadly to social interventions rather than just health and medicine, then the search strategy needs to include a broader set of databases. There are relevant efforts in psychology for example (see Maasen et al. (2020¹)) that could also inform the checklist. I understand if the authors wish to focus this effort only on health and medicine.

In the social sciences, similar issues to those that are outlined in the protocol occur. In addition, many meta-analyses include a large number of studies, and thus key issues center on dependent effect sizes within studies and interpretation of effect size models for heterogeneity. If the authors intend the checklist to have a broader application, then these are a few issues that should be considered in the development of the checklist.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Yes

References

1. Maassen E, van Assen MALM, Nuijten MB, Olsson-Collentine A, et al.: Reproducibility of individual effect sizes in meta-analyses in psychology.PLoS One. 2020; 15 (5): e0233107 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: meta-analysis methods

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Views

22

Reviewer Report 21 Jun 2021

Ram Bajpai, School of Medicine, Keele University, Staffordshire, UK

Approved

https://doi.org/10.5256/f1000research.56375.r87114

Authors have well prepared their systematic review protocol with clarity. I have following minor suggestions to improve the quality of this protocol.

From inclusion criteria it is clear that authors will include SRs of interventions only. However, ... Continue reading

Authors have well prepared their systematic review protocol with clarity. I have following minor suggestions to improve the quality of this protocol.

From inclusion criteria it is clear that authors will include SRs of interventions only. However, if possible, authors should elaborate on not including SR of proportions or SR of observational studies or SR of biomarkers etc. Each set of SRs will require a different type of statistical skills to perform meta-analysis. It also important to mention about whether only pharmacological intervention SRs or non- pharmacological interventions such as educational interventions etc. will also be included in this methodological systematic review.

Did authors think that a statistician included in the author list will have any impact on error as it is possible that SRs with statistician as co-author may have less statistical errors? I think the authors have already thought about this point, however, appreciate authors reflection on this point and if they add few words in the protocol.

If I am not wrong, the planned survey and consensus meeting will follow Delphi survey methodology. I am wondering that authors did not mention this word explicitly or they are following some other equally useful methodology.

I believe that authors have well refined methodology to capture statistical errors. However, in many SRs, authors wrongly choose a meta-analysis model (i.e., fixed-effect vs. random-effect) to pool study estimates and possible other statistics right. How would authors capture such errors? Please elaborate if I have missed this point.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Meta-analysis, Statistical analysis

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 08 Jun 2021

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 1 08 Jun 21	read	read

Ram Bajpai, Keele University, Staffordshire, UK
Therese D. Pigott, Georgia State University, Atlanta, USA

Comments on this article

All Comments(0)

Add a comment

Sign up for content alerts

Browse by related subjects

Back to all reports

Reviewer Report

38 Views

05 Jul 2021 | for Version 1

Therese D. Pigott, School of Public Health and College of Education and Human Development, Georgia State University, Atlanta, GA, USA

38 Views Cite this report Responses(0)

Approved

This work will provide an important contribution to the field, highlighting typical errors that can occur in pairwise meta-analyses of intervention effects. One question I have is the audience of the checklist. The study protocol is situated in meta-analysis for health and medical interventions. The search strategy for relevant systematic reviews for stage one of the process focuses on the health and medical literature, with only the Campbell Collaboration as a source for meta-analyses in the social sciences. If the intent of the checklist is to apply more broadly to social interventions rather than just health and medicine, then the search strategy needs to include a broader set of databases. There are relevant efforts in psychology for example (see Maasen et al. (2020¹)) that could also inform the checklist. I understand if the authors wish to focus this effort only on health and medicine.

In the social sciences, similar issues to those that are outlined in the protocol occur. In addition, many meta-analyses include a large number of studies, and thus key issues center on dependent effect sizes within studies and interpretation of effect size models for heterogeneity. If the authors intend the checklist to have a broader application, then these are a few issues that should be considered in the development of the checklist.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Yes

References

1. Maassen E, van Assen MALM, Nuijten MB, Olsson-Collentine A, et al.: Reproducibility of individual effect sizes in meta-analyses in psychology.PLoS One. 2020; 15 (5): e0233107 PubMed Abstract | Publisher Full Text

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

meta-analysis methods

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

22 Views

21 Jun 2021 | for Version 1

Ram Bajpai, School of Medicine, Keele University, Staffordshire, UK

22 Views Cite this report Responses(0)

Approved

Authors have well prepared their systematic review protocol with clarity. I have following minor suggestions to improve the quality of this protocol.

From inclusion criteria it is clear that authors will include SRs of interventions only. However, if possible, authors should elaborate on not including SR of proportions or SR of observational studies or SR of biomarkers etc. Each set of SRs will require a different type of statistical skills to perform meta-analysis. It also important to mention about whether only pharmacological intervention SRs or non- pharmacological interventions such as educational interventions etc. will also be included in this methodological systematic review.

Did authors think that a statistician included in the author list will have any impact on error as it is possible that SRs with statistician as co-author may have less statistical errors? I think the authors have already thought about this point, however, appreciate authors reflection on this point and if they add few words in the protocol.

If I am not wrong, the planned survey and consensus meeting will follow Delphi survey methodology. I am wondering that authors did not mention this word explicitly or they are following some other equally useful methodology.

I believe that authors have well refined methodology to capture statistical errors. However, in many SRs, authors wrongly choose a meta-analysis model (i.e., fixed-effect vs. random-effect) to pool study estimates and possible other statistics right. How would authors capture such errors? Please elaborate if I have missed this point.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Meta-analysis, Statistical analysis

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

[1] 1. McKenzie JE, Brennan SE: Chapter 12: Synthesizing and presenting findings using other methods. Cochrane Handbook for Systematic Reviews of Interventions. 2019: 321–47.

[2] 2. Richardson M, Garner P, Donegan S: Cluster Randomised Trials in Cochrane Reviews: Evaluation of Methodological and Reporting Practice. PLoS One. 2016; 11(3): e0151818. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Donner A, Klar N: Issues in the meta-analysis of cluster randomized trials. Stat Med. 2002; 21(19): 2971–80. PubMed Abstract | Publisher Full Text

[4] 4. Stedman MR, Curtin F, Elbourne DR, et al.: Meta-analyses involving cross-over trials: methodological issues. Int J Epidemiol. 2011; 40(6): 1732–4. PubMed Abstract | Publisher Full Text

[5] 5. Nolan SJ, Hambleton I, Dwan K: The use and reporting of the cross-over study design in clinical trials and systematic reviews: a systematic assessment. PLoS One. 2016; 11(7): e0159014. PubMed Abstract | Publisher Full Text | Free Full Text

[6] 6. Hrobjartsson A, Gotzsche PC: Unsubstantiated claims of large effects of placebo on pain: serious errors in meta-analysis of placebo analgesia mechanism studies. J Clin Epidemiol. 2006; 59(4): 336–8. discussion 9–41. PubMed Abstract | Publisher Full Text

[7] 7. Jones AP, Remmington T, Williamson PR, et al.: High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol. 2005; 58(7): 741–2. PubMed Abstract | Publisher Full Text

[8] 8. Roba AA, Tefera M, Worku T, et al.: Retraction Note: Application of 4% chlorhexidine to the umbilical cord stump of newborn infants in lower income countries: a systematic review and meta-analysis. Matern Health Neonatol Perinatol. 2020; 6: 4. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Guo LQ, Chen Y, Mi BB, et al.: Retraction Note to: Ambient air pollution and adverse birth outcomes: a systematic review and meta-analysis. J Zhejiang Univ Sci B. 2020; 21(9): 756. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Ezenwa BN, Bello S, Oyenusi EE, et al.: RETRACTED: Galactogogues use Among Mothers With Preterm Births: A Systematic Review and Meta-Analysis. J Hum Lact. 2020; 36(3): NP3–NP14. PubMed Abstract | Publisher Full Text

[11] 11. Page MJ, Altman DG, McKenzie JE, et al.: Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common: a cross-sectional analysis. J Clin Epidemiol. 2018; 95: 7–18. PubMed Abstract | Publisher Full Text

[12] 12. Fisher DJ, Carpenter JR, Morris TP, et al.: Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach? BMJ. 2017; 356. PubMed Abstract | Publisher Full Text | Free Full Text

[13] 13. Cooper H, Hedges LV, Valentine JC: The handbook of research synthesis and meta-analysis: Russell Sage Foundation; 2019.

[14] 14. Schmid CH, White IR, Stijnen T: Introduction to Systematic Review and Meta-Analysis. Handbook of Meta-Analysis. 2020.

[15] 15. Higgins JP, Thomas J, Chandler J, et al.: Cochrane handbook for systematic reviews of interventions. John Wiley & Sons; 2019.

[16] 16. Rennie D: Editorial peer review in biomedical publication: the first international congress. JAMA. 1990; 263(10): 1317. Publisher Full Text

[17] 17. Chauvin A, Ravaud P, Moher D, et al.: Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study. BMC Medicine. 2019; 17(1): 205. PubMed Abstract | Publisher Full Text | Free Full Text

[18] 18. Speich B, Schroter S, Briel M, et al.: Impact of a short version of the CONSORT checklist for peer reviewers to improve the reporting of randomised controlled trials published in biomedical journals: study protocol for a randomised controlled trial. BMJ Open. 2020; 10(3): e035114. PubMed Abstract | Publisher Full Text | Free Full Text

[19] 19. Blanco D, Schroter S, Aldcroft A, et al.: Effect of an editorial intervention to improve the completeness of reporting of randomised trials: a randomised controlled trial. BMJ Open. 2020; 10(5): e036799. PubMed Abstract | Publisher Full Text | Free Full Text

[20] 20. Chauvin A, Moher D, Altman D, et al.: A protocol of a cross-sectional study evaluating an online tool for early career peer reviewers assessing reports of randomised controlled trials. BMJ Open. 2017; 7(9): e017462. PubMed Abstract | Publisher Full Text | Free Full Text

[21] 21. Brown AW, Kaiser KA, Allison DB: Issues with data and analyses: Errors, underlying themes, and potential solutions. Proc Natl Acad Sci U S A. 2018; 115(11): 2563–70. PubMed Abstract | Publisher Full Text | Free Full Text

[22] 22. Page M, Kanukula R, McKenzie J: Search strategy for systematic review of studies evaluating errors in the conduct and interpretation of pairwise meta-analysis. Monash University. Online resource. 2021. [database on the Internet]. Publisher Full Text

[23] 23. Sumsion T: The Delphi technique: an adaptive research tool. Br J Occup Ther. 1998; 61(4): 153–6. Publisher Full Text

[24] 24. Page MJ, Bero L, Kroeger CM, et al.: Investigation of Risk Of Bias due to Unreported and SelecTively included results in meta-analyses of nutrition research: the ROBUST study protocol. F1000Res. 2019; 8(1760): 1760. PubMed Abstract | Publisher Full Text | Free Full Text

[25] 25. Page MJ, Forbes A, Chau M, et al.: Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study. BMJ Open. 2016; 6(4): e011863. PubMed Abstract | Publisher Full Text | Free Full Text

Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol

Abstract

Keywords

1. Introduction

2. Defining the concept of statistical errors

3. Methods

3.1 Contributors

3.2 Systematic review

3.3 Survey

3.4 Virtual meeting

3.5 Piloting

3.6 Dissemination and knowledge translation strategy

4. Conclusion

Data availability

Underlying data

Extended data

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated