Methodological literature on the reporting of systematic reviews of health economic evaluations: a scoping review protocol

Phuong Bich Tran; Joseph Kwon; Anastasios Bastounis; Stavros Petrou; Andrew Booth; PRISMA-EconEval Management Group

doi:10.12688/f1000research.156907.1

Home Browse Methodological literature on the reporting of systematic reviews of...

ALL Metrics

-

Views

-

Downloads

Get PDF

Get XML

Export

▬

✚

Study Protocol

Methodological literature on the reporting of systematic reviews of health economic evaluations: a scoping review protocol

[version 1; peer review: 2 approved, 1 approved with reservations]

Phuong Bich Tran ¹, Joseph Kwon¹, Anastasios Bastounis², Stavros Petrou¹, Andrew Booth², PRISMA-EconEval Management Group

Phuong Bich Tran ¹, Joseph Kwon¹, [...] Anastasios Bastounis², Stavros Petrou¹, Andrew Booth², PRISMA-EconEval Management Group

PUBLISHED 18 Nov 2024

Author details Author details

¹ Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
² Sheffield Centre for Health and Related Research (SCHARR), University of Sheffield, Sheffield, UK

Phuong Bich Tran
Roles: Conceptualization, Investigation, Methodology, Project Administration, Resources, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Joseph Kwon
Roles: Conceptualization, Funding Acquisition, Methodology, Project Administration, Resources, Supervision, Writing – Review & Editing

Anastasios Bastounis
Roles: Conceptualization, Methodology, Resources, Writing – Review & Editing

Stavros Petrou
Roles: Conceptualization, Funding Acquisition, Methodology, Resources, Supervision, Writing – Review & Editing

Andrew Booth
Roles: Conceptualization, Investigation, Methodology, Resources, Software, Supervision, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Systematic reviews of health economic evaluations play a crucial role in informing evidence-based healthcare decisions, yet they lack standardized reporting guidelines. A project has been initiated that aims to extend the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline for systematic reviews of health economic evaluations (PRISMA-EconEval). This scoping review forms a foundation for the PRISMA-EconEval project, aiming to identify, map, and extract candidate reporting items from the methodological literature.

The scoping review will follow the PRISMA Extension for Scoping Reviews (PRISMA-ScR) Checklist and involve comprehensive searches in databases such as PubMed MEDLINE, Embase, and Web of Science, covering the period from 2015 to 2024. Supplementary searching, reference checking and citation searching will target grey literature, overlooked studies and evidence prior to 2015. Inclusion criteria will focus on methodological papers that provide frameworks or recommendations for reporting systematic reviews of health economic evaluations and enhanced case studies that critically discuss methods and reporting structures. The extracted data will be coded and analyzed to produce an initial list of candidate reporting items, structured according to conventional sections of a systematic review (e.g., title, abstract, methods, results). This initial list will be used in the subsequent stages of the project and disseminated through a peer-reviewed publication and presentations at international conferences.

The outcome of this scoping review will significantly contribute to the development of a comprehensive PRISMA-EconEval reporting guideline, aimed at enhancing the transparency, consistency, and quality of systematic reviews of health economic evaluations, and provide an essential tool for authors, editors, peer-reviewers, and stakeholders.

Keywords

reporting, systematic review, health economic evaluation, review protocol

Corresponding author: Phuong Bich Tran

Competing interests: No competing interests were disclosed.

Grant information: This study forms part of the 'Development of new reporting guidance for systematic reviews of health economic evaluations (PRISMA-EE)' project, funded by the National Institute for Health and Care Research (NIHR) Research for Patient Benefit Programme, grant number NIHR206833, hosted at the Nuffield Department of Primary Care Health Sciences, University of Oxford. SP receives support as a NIHR Senior Investigator (NF-SI-0616-10103) and from the NIHR Applied Research Collaboration Oxford and Thames Valley at the Oxford Health NHS Foundation Trust. The funders had no involvement in the development of the protocol.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2024 Tran PB et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Tran PB, Kwon J, Bastounis A et al. Methodological literature on the reporting of systematic reviews of health economic evaluations: a scoping review protocol [version 1; peer review: 2 approved, 1 approved with reservations]. F1000Research 2024, 13:1382 (https://doi.org/10.12688/f1000research.156907.1) First published: 18 Nov 2024, 13:1382 (https://doi.org/10.12688/f1000research.156907.1) Latest published: 18 Nov 2024, 13:1382 (https://doi.org/10.12688/f1000research.156907.1)

Introduction

A systematic review summarizes the available evidence and provides an unbiased, reliable assessment of the current state of knowledge, essential for making evidence-based decisions.¹ The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2020 statement was designed to enhance the reporting of systematic reviews and meta-analyses – primarily of the effects of interventions, though also applicable to systematic reviews and meta-analyses of studies with a different objective – by improving their clarity, consistency, transparency, quality, and overall value.²

Systematic reviews of health economic evaluations differ from other systematic reviews in their approach to reporting key methodological steps, including methods for study selection, data extraction and presentation, and the assessment of quality and relevance. Such reviews consider health economic evaluations, which compare the effects of different healthcare interventions on economic outcomes (e.g. incremental cost-effectiveness ratios, net monetary benefits, etc.). Additionally, they differ in their potential for combining costs, preference-based health-related quality of life outcomes, and cost-effectiveness outcomes across studies.

A literature search in PubMed Central from January 1, 2015 to March 25, 2017 found 202 systematic reviews of health economic evaluations listed within this 27-month period.³ Extending the search to other databases and grey literature and from 2017 to date is likely to increase this number substantially. These reviews lack common standards for reporting quality, highlighting the need for an additional reporting guideline specifically for systematic reviews of health economic evaluations.

The ‘Development of a PRISMA extension for systematic reviews of health economic evaluations (PRISMA-EconEval)’ project was initiated to address this gap. A core international multidisciplinary working group, including PRISMA members, authors, journal editors, and stakeholders, will identify and extract candidate reporting items from various sources. Such sources include this scoping review of the methodological literature on the reporting of systematic reviews of health economic evaluations together with a purposive sample of previous systematic reviews of health economic evaluations. They will refine these items to improve reporting quality. A multi-round online Delphi survey will then evaluate these items with input from academic researchers, specifically health economists (including authors of systematic reviews of health economic evaluations) and systematic review methodologists, journal editors, reporting guideline developers, information specialists, and stakeholders, including health care decision-makers, health research funders, and patient and public representatives. The development of common reference standards for reporting should enhance completeness, transparency and structure in the reporting of systematic reviews of health economic evaluations and generate user-friendly tools for authors, editors, peer-reviewers and stakeholders that facilitate reporting.

As a first step, it is necessary to review the methodological literature on the reporting of systematic reviews of health economic evaluations. This process will help map the current methods, guidance, and recommendations for reporting such reviews and is part of the process of identifying common reference standards for reporting them.

Research objectives

1. To identify and review the methodological literature on the reporting of systematic reviews of health economic evaluations.
2. To summarize and map methods, guidance, and recommendations for reporting such reviews.
3. To identify and extract an initial dataset of candidate reporting items.

Method

Protocol

The scoping review forms part of the larger PRISMA-EconEval project. The protocol for the scoping review follows the Preferred Reporting Items for Systematic Review and Meta -Analysis Protocols (PRISMA-P) 2015 Checklist,⁴ with the protocol and PRISMA-P checklist deposited on the Open Science Framework.⁵ We will report the scoping review in accordance with the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist.⁶

The scoping review will be conducted in various stages as shown in Figure 1 and will aim to identify an initial list of candidate reporting items. This initial list of reporting items will have generic relevance to systematic reviews of health economic evaluations regardless of the types of health economic evaluation (e.g., cost-minimization analysis, cost-benefit analysis, cost-effectiveness analysis, cost-utility analysis, cost-consequences analysis) and the vehicles for health economic evaluation (e.g., economic evaluations based on randomized controlled trials, economic evaluations based on observational studies with patient-level data, decision-analytic models) adopted by the individual studies they cover. Specific reporting items relevant to different categories of health economic evaluations may also be identified. The initial list will then be enhanced by findings from an additional sample of published systematic reviews of health economic evaluations. After refinement by the PRISMA-EconEval Management Group, this list will inform the multi-round online Delphi survey. Detailed plans for subsequent stages (post-scoping review) will be outlined in the comprehensive PRISMA-EconEval project protocol.

Figure 1. Stages of the PRISMA-EconEval project.

The scoping review will focus on two types of articles (either published in journals or elsewhere):

• Methodological papers: Articles that describe methods, frameworks, guidance for conduct, or recommendations on the reporting of systematic reviews of health economic evaluations.
• Enhanced case studies: Systematic reviews of health economic evaluations that include a minimum of two paragraphs that discuss methods, conceptual approaches, limitations (e.g., items not analyzed/reported in the review), and/or thorough categorization and reporting of health economic evaluation features that can inform the initial list of candidate reporting items.

Conditions or domains being studied

We used the ECLIPSE Framework to help in framing the scope and objectives of the scoping review ( Table 1).⁷

Table 1. The ECLIPSE framework.

Element	Definition	Description
Expectation	What are you looking to improve or change? What is the information going to be used for?	We aim to collate and summarize methods, guidance, and recommendations for the reporting of systematic reviews of health economic evaluations. This is an initial step in the development of a new reporting guideline for systematic reviews of health economic evaluations.
Client group	Who is the service or policy aimed at?	Authors, editors, peer-reviewers and stakeholders that conduct or facilitate the reporting of systematic reviews of health economic evaluations.
Location	Where is the service or policy located?	Unrestricted
Impact	What is the change in service or policy that the researcher is investigating?	Findings from the scoping review will form the initial dataset of candidate reporting items, which is a key step in developing the PRISMA-EconEval guideline. The development of a common reference standard for reporting should enhance completeness, transparency and structure in the reporting of systematic reviews of health economic evaluations.
Professionals	Who is involved in providing or improving the service or policy?	PRISMA-EconEval Management Group, Independent Advisory Group, Delphi Panelists, Patient and Public Involvement Group, Pilot End-Users
Service	What kind of service or policy is this?	The development of a reporting guideline

Expected timeline for scoping review

August 2024 – December 2024.

Search strategy

We will conduct searches in PubMed MEDLINE, Embase, and Web of Science for the period from 2015 to 2024. Relevant papers published prior to this timeframe will be identified through supplementary searching, reference checking and citation searching.

For the grey literature, we will identify and agree upon a list of 10-12 websites of health technology assessment agencies or health economics associations that may publish methods papers, guidance, and recommendations for reporting systematic reviews of health economic evaluations.

Search terms and strings

An experienced information specialist (ABo) will search databases of the published literature, applying guidance on filters at the intersection of “systematic reviews” and “economic evaluations” to develop and pilot our search strategies, including data sources, date limits, search terms, and syntax (Box 1).

Box 1. Search string examples for PubMed.

“systematic review of economic evaluations”[tiab:~0] OR “systematic review of cost effectiveness”[tiab:~0] OR “systematic review”[PT] AND (economic*[Ti] OR (“economic evaluation*” OR cost [Ti] OR costs [Ti] OR “cost effectiveness” OR “cost benefit analysis” OR “Cost Utility Analysis” OR “Cost-Benefit Analysis”)

“systematic reviews of economic evaluations”[tiab:~0] OR “systematic reviews of cost effectiveness”[tiab:~0] OR ((“Systematic Reviews as Topic” OR “Review Literature As Topic”) AND (economics [SH] OR economic*[Ti] OR cost [Ti] OR costs [Ti] OR “cost effectiveness” OR “cost benefit analysis” OR “Cost utility Analysis” OR “Cost-Benefit Analysis”)) OR (“systematic reviews” AND (“economic evaluation*” OR cost [Ti] OR costs [Ti] OR “cost effectiveness” OR “cost benefit analysis” OR “Cost utility Analysis” OR “Cost-Benefit Analysis”) AND (Methods OR methodology OR methodological OR checklist* OR guideline*)) OR ((“Cost-Benefit Analysis” AND (Methods OR methodology OR methodological OR checklist* OR guideline*) AND (“Systematic Reviews as Topic” OR “Review Literature As Topic”))

For the grey literature, the scoping review team will search the websites of health technology assessment agencies and health economics associations, using similar search terms.

Supplementary search methods

Recognizing that methodological considerations are not always adequately reported in the titles and abstracts of systematic reviews of health economic evaluations, the research team will use innovative supplementary search methods to access the contents of full text articles and establish their eligibility for inclusion. Scite (open access alternative — Consensus) and Litsense will be interrogated for “systematic reviews of (health) economic evaluations”. Google Scholar will be searched using targeted search terms via the free open source Publish or Perish search tool to access full text contents of high currency and specificity. We will also use BASE,⁸ one of the largest search engines for academic web resources, which captures both academic and grey literature that may have been missed. Finally, a list of 10-12 known “index” items of high relevance to the review question will be used as the starting point for both backwards citation chaining (reference checking) and forwards citation chaining (citation chasing) using the Citation Chaser open-source tool.

Inclusion and exclusion criteria

For methodological papers, we will include articles that describe methods, frameworks, guidance, or recommendations on the reporting of systematic reviews of health economic evaluations. For enhanced case studies, we will include systematic reviews of health economic evaluations that contain at least two paragraphs critically discussing methods, conceptual approaches, limitations (e.g., items that were not reported but should have been), and/or thorough categorization and reporting of health economic evaluation features that can inform the initial list of candidate reporting items. Both types of articles published in English will be included without date restriction. We will exclude conference abstracts and proceedings, except for those that specifically focus on reporting methodologies of systematic reviews of health economic evaluations. For these, we will conduct additional searches and contact the authors (if necessary) to determine if a full-text publication is available. If a full-text version cannot be identified, they will be excluded.

Article screening and selection

References for the articles retrieved will be uploaded to Covidence⁹ (open access alternative — Rayyan) for deduplication and initial screening (Phase One) and, subsequently, for full-text screening and reference management (Phase Two).

The eligibility of each identified study will be assessed by two of the five reviewers (ABa, ABo, JK, PT, SP) independently. A two-stage standard protocol will be developed and followed via checklists for each of the two phases. The checklists will be piloted on 100 random articles and screening criteria adapted accordingly.

Phase One screening (Title and abstract screening): The following checklist will be used for the first screening of articles after deduplication ( Table 2). All articles will be classified accordingly. Each independently classified article by every two reviewers will be automatically compared by Covidence. Disagreements between reviewers will be resolved through consensus or by referral to a third reviewer.

Table 2. Phase One screening checklist.

Q	Question	Answer	Action following answer	Comments/examples
1	Based on title/abstract, is the article likely to focus on reporting methodologies for systematic reviews/evidence syntheses of health economic evaluations?	Yes	Classify as “Retained phase 1, methodological paper”	The primary focus of the article should be on methods, frameworks, guidance, or recommendations on the reporting of systematic reviews of health economic evaluations.
		No	Go to Q2
		Maybe	Go to Q2
2	Based on title/abstract, is the article a systematic review of health economic evaluations: i.e., a ‘worked example’? Alternatively, is it likely to focus on such reviews, or at minimum, on health economic evaluations?	Yes	Go to Q3	The article should at minimum, focus on full health economic evaluations or systematic reviews of such studies. Review of reviews may also be helpful. Partial health economic evaluations (e.g., cost analyses, utility elicitation studies) should be excluded.
		No	Classify as “Exclude”
		Maybe	Go to Q3
3	Does the title/abstract discuss issues relevant to or contain useful information regarding the reporting of systematic reviews of health economic evaluations?	Yes	Classify as “Retained phase 1, enhanced case study”	In the case of ‘worked examples’, the abstract may show hints of discussions related to the reporting of systematic reviews of health economic evaluations. (e.g., Studies demonstrated variation or heterogeneity in reporting; XX studies did not include the perspective of the analysis, and YY studies failed to specify the choice of discount rate.) The abstract may also include thorough categorizations that could inform the list of candidate reporting items. (e.g., Cost-effectiveness was estimated using partitioned survival models in XX studies and state-transition models in YY studies. Across studies, XX reported utility values by health state and YY reported disease-specific HRQoL.)
		No	Classify as “Exclude”
		Maybe	Classify as “Retained phase 1, potential enhanced case study”

Phase Two screening (Full-text screening): For all articles classified as “Retained Phase 1”, the full article will be read and further classified as below ( Table 3). Disagreements between reviewers will be resolved through consensus or by referral to a third reviewer.

Table 3. Phase Two screening checklist.

Q	Question	Answer	Action following answer	Comments/examples
1	Does the article focus on reporting methodologies for systematic reviews/evidence syntheses of health economic evaluations?	Yes	Classify as “Include, methodological paper”	The primary focus of the article should be on methods, frameworks, guidance, or recommendations on the reporting of systematic reviews of health economic evaluations.
		No	Go to Q2
		No	Classify as “Exclude”
2	Does the article contain at least two paragraphs critically discussing methods, conceptual approaches, limitations and/or recommendations for the structuring and reporting of systematic reviews of health economic evaluations. Alternatively, does it contain useful information regarding the reporting of systematic reviews of health economic evaluations?	Yes	Classify as “Include, enhanced case study”	In the case of ‘worked examples’, the article may (e.g., in the discussion section) discuss issues relating to the reporting of systematic reviews of health economic evaluations. In the result section, they may also discuss items that were not reported but should have been. (e.g., Studies demonstrated variation or heterogeneity in reporting; XX studies did not include the perspective of the analysis, and YY studies failed to specify the choice of discount rate.) The article may also include thorough categorizations that could inform the list of candidate reporting items. (e.g., Cost-effectiveness was estimated using partitioned survival models in XX studies and state-transition models in YY studies. Across studies, XX reported utility values by health state and YY reported disease-specific HRQoL.)
2		No	Classify as “Exclude”

Data extraction and coding

Data from studies that fulfill the inclusion criteria will be entered (by ABa, ABo, JK, PT, SP) onto an Excel form. Data to be extracted will include: article PMID number, title of paper, authors, year of publication, journal or source, type of literature (i.e., published scientific article, grey literature), and type of paper (i.e., methodological paper, enhanced case study). Recommendations from included studies on the methods and reporting of systematic reviews of health economic evaluations will be extracted and organized by codes and sub-codes. The coding will be conducted using NVivo V.1.7¹⁰ (open access alternative — QualCoder). The reviewers will work together using a shared open codebook. All domains described in the included studies will be gathered and tabulated. Conventional sections and sub-sections of a systematic review as informed by the PRISMA guideline (i.e., title, abstract, introduction, methods, results, discussion, other information) will be used to inform the coding by which the identified themes and topics are organized. The extraction process will follow a pilot round of coding, continual refinement and revision of the codebook, and frequent consensus meetings.

Data analysis and reporting

The objective of the scoping review is to produce an initial list of recommended reporting items for systematic reviews of health economic evaluations, to be used in subsequent stages of the PRISMA-EconEval project. Where necessary, items will be renamed/rephrased, similar items will be combined, and items will be split if they cover two or more distinct topics.

The initial list of candidate reporting items identified by the PRISMA-EconEval Management Group will be arranged according to the conventional sections of a systematic review: title, abstract, introduction, methods, results, discussion, and other information. This structure broadly aligns with other extensions to the PRISMA reporting guidelines. The initial list of candidate reporting items will have generic relevance to systematic reviews regardless of the types of health economic evaluation or the vehicles for health economic evaluation adopted by the individual studies they cover. We will also include specific reporting items relevant to different categories of health economic evaluations in this and subsequent stages of the PRISMA-EconEval project.

Dissemination

We plan to disseminate the findings of the scoping review by submitting the resulting manuscript to a suitable peer-reviewed journal and presenting the key findings at the 2025 International Health Economics Association Congress. The initial list of candidate reporting items will be used in the subsequent stages of the project; i.e., review of sample papers and the Delphi surveys.

Study status

The protocol has been registered on the EQUATOR Network database¹¹ and the Open Science Framework.⁵ The database searches have been conducted and pilot screening will soon take place.

Ethical considerations

Ethical approval is not required for this aspect of the PRISMA-EconEval project as it does not involve individual or personal data.

Author contributions

The review was conceptualized collectively by the team (PT, JK, ABa, SP, ABo). PT developed the initial draft manuscript and created visualizations, with reviews and edits by the team. ABo and ABa provided expertise on the review methodology, databases and search strategy. PT was responsible for the revision and submission of the manuscript and is the guarantor of the protocol. SP, JK, ABo provided supervision throughout the entire process. All authors have reviewed and approved the final version of the manuscript.

List of members of the PRISMA-EconEval Management Group

1. Prof Stavros Petrou, Nuffield Department of Primary Care Health Sciences, University of Oxford, United Kingdom
2. Dr Joseph Kwon, Nuffield Department of Primary Care Health Sciences, University of Oxford, United Kingdom
3. Dr Phuong Bich Tran, Nuffield Department of Primary Care Health Sciences, University of Oxford, United Kingdom
4. Prof Andrew Booth, Sheffield Centre for Health and Related Research (SCHARR), University of Sheffield, United Kingdom
5. Dr Anastasios Bastounis, Sheffield Centre for Health and Related Research (SCHARR), University of Sheffield, United Kingdom
6. Prof Sophie Staniszewska, RCN Research Institute, Warwick Medical School, University of Warwick, United Kingdom
7. Mr Richard Grant, Patient and Public Involvement Representative, United Kingdom
8. Prof Sally Hopewell, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, United Kingdom
9. Dr Matthew Page, Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Australia

Data availability

Underlying data

No underlying data is associated with this article.

Extended data

Reporting guideline.

Open Science Framework. [Development of new reporting guidance for systematic reviews of health economic evaluations (PRISMA-EconEval)] DOI: https://doi.org/10.17605/OSF.IO/FSPG9.⁵

This project contains the following data:

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

References

1. Moosapour H, Saeidifard F, Aalaa M, et al.: The rationale behind systematic reviews in clinical medicine: a conceptual framework. J. Diabetes Metab. Disord. 2021 Apr 8; 20(1): 919–929. PubMed Abstract | Publisher Full Text | Free Full Text
2. Page MJ, McKenzie JE, Bossuyt PM, et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021 Mar 29; 372: n71. PubMed Abstract | Publisher Full Text | Free Full Text
3. Luhnen M, Prediger B, Neugebauer EAM, et al.: Systematic reviews of health economic evaluations: A structured analysis of characteristics and methods applied. Res. Synth. Methods. 2019 Jun; 10(2): 195–206. PubMed Abstract | Publisher Full Text
4. Shamseer L, Moher D, Clarke M, et al.: Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015 Jan 2; 349: g7647. Publisher Full Text
5. Kwon J, Petrou S, Tran PB, et al.: Development of new reporting guidance for systematic reviews of health economic evaluations (PRISMA-EconEval). OSF. 2024 Jul 30 [cited 2024 Sep 26]. Publisher Full Text
6. Tricco AC, Lillie E, Zarin W, et al.: PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann. Intern. Med. 2018 Oct 2; 169(7): 467–473. PubMed Abstract | Publisher Full Text
7. Wildridge V, Bell L: How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Inf. Libr. J. 2002 Jun 1; 19(2): 113–115. PubMed Abstract | Publisher Full Text
8. Bielefeld University Library: BASE - Bielefeld Academic Search Engine.[cited 2024 Oct 10]. Reference Source
9. Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation; 2024. Reference Source
10. QSR International: NVIVO Release 1.7.2022. Reference Source
11. EQUATOR Network: Reporting guidelines under development for systematic reviews.2024 [cited 2024 Oct 10]. Reference Source

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 18 Nov 2024

Author details Author details

¹ Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
² Sheffield Centre for Health and Related Research (SCHARR), University of Sheffield, Sheffield, UK

Phuong Bich Tran
Roles: Conceptualization, Investigation, Methodology, Project Administration, Resources, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Joseph Kwon
Roles: Conceptualization, Funding Acquisition, Methodology, Project Administration, Resources, Supervision, Writing – Review & Editing

Anastasios Bastounis
Roles: Conceptualization, Methodology, Resources, Writing – Review & Editing

Stavros Petrou
Roles: Conceptualization, Funding Acquisition, Methodology, Resources, Supervision, Writing – Review & Editing

Andrew Booth
Roles: Conceptualization, Investigation, Methodology, Resources, Software, Supervision, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

This study forms part of the 'Development of new reporting guidance for systematic reviews of health economic evaluations (PRISMA-EE)' project, funded by the National Institute for Health and Care Research (NIHR) Research for Patient Benefit Programme, grant number NIHR206833, hosted at the Nuffield Department of Primary Care Health Sciences, University of Oxford. SP receives support as a NIHR Senior Investigator (NF-SI-0616-10103) and from the NIHR Applied Research Collaboration Oxford and Thames Valley at the Oxford Health NHS Foundation Trust. The funders had no involvement in the development of the protocol.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (1)

version 1

Published: 18 Nov 2024, 13:1382

https://doi.org/10.12688/f1000research.156907.1

Copyright

© 2024 Tran PB et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

0

SEE MORE DETAILS

CITE

how to cite this article

Tran PB, Kwon J, Bastounis A et al. Methodological literature on the reporting of systematic reviews of health economic evaluations: a scoping review protocol [version 1; peer review: 2 approved, 1 approved with reservations]. F1000Research 2024, 13:1382 (https://doi.org/10.12688/f1000research.156907.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 1

VERSION 1

PUBLISHED 18 Nov 2024

Views

5

Reviewer Report 26 Feb 2025

Marta Trapero-Bertran, Department of Economics and Business, Biomedical Research Institute of Lleida, University of Lleida, Advisory Committee on the Financing of Pharmaceutical (CAPF), Lleida, Spain

Approved

https://doi.org/10.5256/f1000research.172286.r363770

I would like to congratulate you on this outstanding piece of work. Your article is highly engaging, well-written, and clearly articulated, making it a valuable contribution to the academic field.
The only comment I would like to make relates ... Continue reading

I would like to congratulate you on this outstanding piece of work. Your article is highly engaging, well-written, and clearly articulated, making it a valuable contribution to the academic field.
The only comment I would like to make relates to the search terms used in your bibliographic review. Rather than concatenating all the suggested search terms as an example of a search strategy in Medline, it might be more efficient to use the * symbol to avoid explicitly listing all possible variations of economic evaluation types. In this regard, I would suggest something like the following search strategy:
"systemat* rev*" AND ("cost*" OR "econ* eval*") AND ("checklist*" OR "method" OR "anal*")**
This adjustment could help broaden the search results while maintaining specificity, and shortening search terms used.
Once again, congratulations on this excellent article, and thank you for your valuable contribution.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Partly
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Health Economics and Economic Evaluation

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Views

5

Reviewer Report 20 Feb 2025

Vicki Brown, Deakin Health Economics, Institute for Health Transformation, Deakin University, Geelong, Australia

Approved

https://doi.org/10.5256/f1000research.172286.r363774

This paper presents the protocol for a scoping review of systematic reviews of health economic evaluations, with the aim of informing the development of a PRISMA "add-on" for reporting of SRs of health economic evaluations. The manuscript is clear and ... Continue reading

This paper presents the protocol for a scoping review of systematic reviews of health economic evaluations, with the aim of informing the development of a PRISMA "add-on" for reporting of SRs of health economic evaluations. The manuscript is clear and well written. The methods proposed are sound. I have no comments or edits.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Health economics

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Views

6

Reviewer Report 07 Jan 2025

Ghislaine A.P.G. van Mastrigt, Department of Health Services Research, Care and Public Health Research Institute (CAPHRI), Maastricht University, Maastricht, The Netherlands

Approved with Reservations

https://doi.org/10.5256/f1000research.172286.r347340

This study aims to investigate the existing literature through a scoping review to develop an alternative to the well-established PRISMA reporting guidelines, namely one to improve the reporting quality of health economic evaluations. Overall, this is a very well-written ... Continue reading

This study aims to investigate the existing literature through a scoping review to develop an alternative to the well-established PRISMA reporting guidelines, namely one to improve the reporting quality of health economic evaluations. Overall, this is a very well-written and clear protocol for performing this research. However, the authors were given a few suggestions to improve this scoping review protocol.
Regarding the questions of the review report form:

Is the rationale for, and objectives of, the study clearly described?
Partly
The authors report the three research objectives:

1. To identify and review the methodological literature on reporting systematic reviews of health economic evaluations.
2. To summarize and map methods, guidance, and recommendations for reporting such reviews.
3. To identify and extract an initial dataset of candidate reporting items.

For objective 1, I would not solely focus on ‘reporting’ whether you can identify all the relevant literature; I would rephrase it as follows: To identify and review the methodological literature of systematic reviews of health economic evaluations to improve the reporting quality of these studies.
For objective 2: it would rephrase it as follows: To summarize and map methods, guidance, and recommendations for methodological literature of systematic reviews of health economic evaluations.
For objective 3: to identify and extract an initial dataset of candidate items to develop an extension to PRISMA guidelines for Health economic evaluations

In summary, what aspects do you consider as part of reporting economic evaluations? This could be specified for the objectives, but this is also needed for the protocol.

Is the study design appropriate for the research question?
Yes
The authors propose a scoping review to investigate the existing literature on this topic, I agree this is an appropriate research design to investigate this research question.

Are sufficient details of the methods provided to allow replication by others?
Partly
Regarding the step of the selection of the databases: They state: ‘’ We will conduct searches in PubMed MEDLINE, Embase, and Web of Science for the period from 2015 to 2024’’. However, no rationale behind the choices of these bibliographic databases is given; for economic topics; Econlit could also be a relevant database to check.

Other topics
General topic: study designs
For the scoping review but also the PRISMA-EE development; what kind of types of economic evidence do the authors want to cover in their reporting guidelines? only full economic evaluations: like cost-utility or cost-effectiveness analyses or also partial economic evaluations like; cost of illness studies, resource use etc. This is explained in the phase one checklist but should also be more clearly mentioned in the protocol.
It is highly recommended to start the study after the registration of the protocol; so please adapt the start date of review to January 2025.
Sections: search strategy/search terms and strings
This section could be improved by adding additional information on search strategy development and databases selected (for details see comments below).
It is highly recommended to report the search strings for each of the bibliographic databases which will be used to identify relevant studies. In addition, it is also highly recommended to use validated and predefined search strings as much as possible. The ISSG website is a recommended source to look for, especially for the economic topics (table needs further specification). Furthermore, more details could be given on how the search strategies were developed. For this the Prisma extension ‘’PRESS’’ is very helpful.
The search strategies listed in box 1 do not match the study objectives and could be made more specific.
Details of the 10-12 websites of health technology assessment agencies or health economics associations could be added in an appendix.
Please also specify the 10-12 ‘’index items’’ mentioned to be used as starting points for backward and forward citation chasing. ‘’Citation chasing’’ is mentioned twice, once briefly at the beginning of the section and more detail at the end, one of them could be deleted.
Section: Inclusion and exclusion criteria
Details regarding the type of economic evaluation included, approaches used (e.g. model/trial-based), as well as including full or also partial economic evaluations should be mentioned in the list of eligibility criteria.
Section: Article screening and selection
Will you use Ryan or Covidence to handle the study results?
In the sentence: ‘’The checklists will be piloted on 100 random articles and screening criteria adapted accordingly’’. You could mention that you use two separate checklists for screening and selection.
Add a sentence on the approach you will use to contact authors to obtain full-text papers if only abstracts are available.
Section: data extraction and coding
Will you use NVivo V.1.7¹⁰ or QualCoder for coding?
A draft of the data extraction sheets could be added to the protocol.
In addition a list of example general or specific candidate items?
What do you mean by the last part of this sentence: ‘’We will also include specific reporting items relevant to different categories of health economic evaluations in this and subsequent stages of the PRISMA-EconEval project’’? Could you please explain it?

Are the datasets presented in a usable and accessible format?
Not applicable

Is the rationale for, and objectives of, the study clearly described?

Partly
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Health Economist, Methodologist, and Epidemiologist, in the field of systematic reviews of economic evaluations, quality asssessment and reporting of economic evaluations

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 18 Nov 2024

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3
Version 1 18 Nov 24	read	read	read

Ghislaine A.P.G. van Mastrigt, Maastricht University, Maastricht, The Netherlands
Vicki Brown, Deakin Health Economics, Institute for Health Transformation, Deakin University, Geelong, Australia
Marta Trapero-Bertran, University of Lleida, Advisory Committee on the Financing of Pharmaceutical (CAPF), Lleida, Spain

Comments on this article

All Comments(0)

Add a comment

Sign up for content alerts

Browse by related subjects

Back to all reports

Reviewer Report

5 Views

26 Feb 2025 | for Version 1

Marta Trapero-Bertran, Department of Economics and Business, Biomedical Research Institute of Lleida, University of Lleida, Advisory Committee on the Financing of Pharmaceutical (CAPF), Lleida, Spain

5 Views Cite this report Responses(0)

Approved

I would like to congratulate you on this outstanding piece of work. Your article is highly engaging, well-written, and clearly articulated, making it a valuable contribution to the academic field.
The only comment I would like to make relates to the search terms used in your bibliographic review. Rather than concatenating all the suggested search terms as an example of a search strategy in Medline, it might be more efficient to use the * symbol to avoid explicitly listing all possible variations of economic evaluation types. In this regard, I would suggest something like the following search strategy:
"systemat* rev*" AND ("cost*" OR "econ* eval*") AND ("checklist*" OR "method" OR "anal*")**
This adjustment could help broaden the search results while maintaining specificity, and shortening search terms used.
Once again, congratulations on this excellent article, and thank you for your valuable contribution.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Partly
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Health Economics and Economic Evaluation

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

5 Views

20 Feb 2025 | for Version 1

Vicki Brown, Deakin Health Economics, Institute for Health Transformation, Deakin University, Geelong, Australia

5 Views Cite this report Responses(0)

Approved

This paper presents the protocol for a scoping review of systematic reviews of health economic evaluations, with the aim of informing the development of a PRISMA "add-on" for reporting of SRs of health economic evaluations. The manuscript is clear and well written. The methods proposed are sound. I have no comments or edits.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Health economics

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

6 Views

07 Jan 2025 | for Version 1

Ghislaine A.P.G. van Mastrigt, Department of Health Services Research, Care and Public Health Research Institute (CAPHRI), Maastricht University, Maastricht, The Netherlands

6 Views Cite this report Responses(0)

Approved With Reservations

This study aims to investigate the existing literature through a scoping review to develop an alternative to the well-established PRISMA reporting guidelines, namely one to improve the reporting quality of health economic evaluations. Overall, this is a very well-written and clear protocol for performing this research. However, the authors were given a few suggestions to improve this scoping review protocol.
Regarding the questions of the review report form:

Is the rationale for, and objectives of, the study clearly described?
Partly
The authors report the three research objectives:

1. To identify and review the methodological literature on reporting systematic reviews of health economic evaluations.
2. To summarize and map methods, guidance, and recommendations for reporting such reviews.
3. To identify and extract an initial dataset of candidate reporting items.

For objective 1, I would not solely focus on ‘reporting’ whether you can identify all the relevant literature; I would rephrase it as follows: To identify and review the methodological literature of systematic reviews of health economic evaluations to improve the reporting quality of these studies.
For objective 2: it would rephrase it as follows: To summarize and map methods, guidance, and recommendations for methodological literature of systematic reviews of health economic evaluations.
For objective 3: to identify and extract an initial dataset of candidate items to develop an extension to PRISMA guidelines for Health economic evaluations

In summary, what aspects do you consider as part of reporting economic evaluations? This could be specified for the objectives, but this is also needed for the protocol.

Is the study design appropriate for the research question?
Yes
The authors propose a scoping review to investigate the existing literature on this topic, I agree this is an appropriate research design to investigate this research question.

Are sufficient details of the methods provided to allow replication by others?
Partly
Regarding the step of the selection of the databases: They state: ‘’ We will conduct searches in PubMed MEDLINE, Embase, and Web of Science for the period from 2015 to 2024’’. However, no rationale behind the choices of these bibliographic databases is given; for economic topics; Econlit could also be a relevant database to check.

Other topics
General topic: study designs
For the scoping review but also the PRISMA-EE development; what kind of types of economic evidence do the authors want to cover in their reporting guidelines? only full economic evaluations: like cost-utility or cost-effectiveness analyses or also partial economic evaluations like; cost of illness studies, resource use etc. This is explained in the phase one checklist but should also be more clearly mentioned in the protocol.
It is highly recommended to start the study after the registration of the protocol; so please adapt the start date of review to January 2025.
Sections: search strategy/search terms and strings
This section could be improved by adding additional information on search strategy development and databases selected (for details see comments below).
It is highly recommended to report the search strings for each of the bibliographic databases which will be used to identify relevant studies. In addition, it is also highly recommended to use validated and predefined search strings as much as possible. The ISSG website is a recommended source to look for, especially for the economic topics (table needs further specification). Furthermore, more details could be given on how the search strategies were developed. For this the Prisma extension ‘’PRESS’’ is very helpful.
The search strategies listed in box 1 do not match the study objectives and could be made more specific.
Details of the 10-12 websites of health technology assessment agencies or health economics associations could be added in an appendix.
Please also specify the 10-12 ‘’index items’’ mentioned to be used as starting points for backward and forward citation chasing. ‘’Citation chasing’’ is mentioned twice, once briefly at the beginning of the section and more detail at the end, one of them could be deleted.
Section: Inclusion and exclusion criteria
Details regarding the type of economic evaluation included, approaches used (e.g. model/trial-based), as well as including full or also partial economic evaluations should be mentioned in the list of eligibility criteria.
Section: Article screening and selection
Will you use Ryan or Covidence to handle the study results?
In the sentence: ‘’The checklists will be piloted on 100 random articles and screening criteria adapted accordingly’’. You could mention that you use two separate checklists for screening and selection.
Add a sentence on the approach you will use to contact authors to obtain full-text papers if only abstracts are available.
Section: data extraction and coding
Will you use NVivo V.1.7¹⁰ or QualCoder for coding?
A draft of the data extraction sheets could be added to the protocol.
In addition a list of example general or specific candidate items?
What do you mean by the last part of this sentence: ‘’We will also include specific reporting items relevant to different categories of health economic evaluations in this and subsequent stages of the PRISMA-EconEval project’’? Could you please explain it?

Are the datasets presented in a usable and accessible format?
Not applicable

Is the rationale for, and objectives of, the study clearly described?

Partly
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Health Economist, Methodologist, and Epidemiologist, in the field of systematic reviews of economic evaluations, quality asssessment and reporting of economic evaluations

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Respond to this report

Responses (0)

[1] 1. Moosapour H, Saeidifard F, Aalaa M, et al.: The rationale behind systematic reviews in clinical medicine: a conceptual framework. J. Diabetes Metab. Disord. 2021 Apr 8; 20(1): 919–929. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Page MJ, McKenzie JE, Bossuyt PM, et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021 Mar 29; 372: n71. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Luhnen M, Prediger B, Neugebauer EAM, et al.: Systematic reviews of health economic evaluations: A structured analysis of characteristics and methods applied. Res. Synth. Methods. 2019 Jun; 10(2): 195–206. PubMed Abstract | Publisher Full Text

[4] 4. Shamseer L, Moher D, Clarke M, et al.: Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015 Jan 2; 349: g7647. Publisher Full Text

[5] 5. Kwon J, Petrou S, Tran PB, et al.: Development of new reporting guidance for systematic reviews of health economic evaluations (PRISMA-EconEval). OSF. 2024 Jul 30 [cited 2024 Sep 26]. Publisher Full Text

[6] 6. Tricco AC, Lillie E, Zarin W, et al.: PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann. Intern. Med. 2018 Oct 2; 169(7): 467–473. PubMed Abstract | Publisher Full Text

[7] 7. Wildridge V, Bell L: How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Inf. Libr. J. 2002 Jun 1; 19(2): 113–115. PubMed Abstract | Publisher Full Text

[8] 8. Bielefeld University Library: BASE - Bielefeld Academic Search Engine.[cited 2024 Oct 10]. Reference Source

[9] 9. Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation; 2024. Reference Source

[10] 10. QSR International: NVIVO Release 1.7.2022. Reference Source

[11] 11. EQUATOR Network: Reporting guidelines under development for systematic reviews.2024 [cited 2024 Oct 10]. Reference Source

Methodological literature on the reporting of systematic reviews of health economic evaluations: a scoping review protocol

Abstract

Keywords

Introduction

Method

Protocol

Figure 1. Stages of the PRISMA-EconEval project.

Conditions or domains being studied

Table 1. The ECLIPSE framework.

Expected timeline for scoping review

Search strategy

Search terms and strings

Box 1. Search string examples for PubMed.

Supplementary search methods

Inclusion and exclusion criteria

Article screening and selection

Table 2. Phase One screening checklist.

Table 3. Phase Two screening checklist.

Data extraction and coding

Data analysis and reporting

Dissemination

Study status

Ethical considerations

Author contributions

Data availability

Underlying data

Extended data

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated