Searching for non-English literature may be unnecessary for HTA Reports

Elke Hausner; Sibylle Sturtz; Sandra Molnar; Lisa Schell; Wiebke Sieben; Stefan Sauerland

doi:10.12688/f1000research.151365.1

Home Browse Searching for non-English literature may be unnecessary for HTA Reports

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Searching for non-English literature may be unnecessary for HTA Reports

[version 1; peer review: 2 approved with reservations, 1 not approved]

Elke Hausner ¹, Sibylle Sturtz², Sandra Molnar³, Lisa Schell³, Wiebke Sieben², Stefan Sauerland⁴

Elke Hausner ¹, Sibylle Sturtz², [...] Sandra Molnar³, Lisa Schell³, Wiebke Sieben², Stefan Sauerland⁴

PUBLISHED 04 Oct 2024

Author details Author details

¹ Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, Im Mediapark 8, 50670, Germany
² Department of Medical Biometry, Institute for Quality and Efficiency in Health Care, Cologne, Im Mediapark 8, 50670, Germany
³ Former employee: Department of Non-Drug Interventions, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany
⁴ Department of Non-Drug Interventions, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

Elke Hausner
Roles: Conceptualization, Methodology, Project Administration, Writing – Original Draft Preparation

Sibylle Sturtz
Roles: Data Curation, Formal Analysis, Methodology, Writing – Original Draft Preparation

Sandra Molnar
Roles: Data Curation, Methodology, Writing – Original Draft Preparation

Lisa Schell
Roles: Conceptualization, Data Curation, Methodology, Project Administration, Writing – Original Draft Preparation

Wiebke Sieben
Roles: Data Curation, Formal Analysis, Methodology, Writing – Review & Editing

Stefan Sauerland
Roles: Conceptualization, Supervision, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Future of Research (FoR) collection.

Abstract

Background

Health technology assessment (HTA) reports are based on comprehensive information retrieval. Current standards discourage the use of search restrictions, such as publication date and language. Given limited resources, it was unclear whether the effort invested in screening and translating studies published in languages other than English provided relevant additional information compared with the inclusion of English-language publications alone. We therefore analysed the impact of non-English publications on the conclusions of HTA reports produced by the German HTA agency, the Institute for Quality and Efficiency in Health Care (IQWiG).

Methods

We determined whether non-English publications were included in all HTA reports on non-drug interventions and on selected drug interventions (search period: 01/2011 to 08/2018). If at least one non-English publication was included, we assessed for each endpoint whether or not the exclusion of non-English publications changed the conclusion. If a non-English publication did not contain information relevant to the HTA report, we classified the publication as “not relevant”.

Results

Of 70 HTA reports, 38 (54%) included 128 non-English publications. In 4 reports (6%) with 50 endpoints investigated in 39 PICO questions, the exclusion of a total of 10 non-English publications led to a change in the conclusions for 13 endpoints (8 PICO questions). This was largely due to the fact that in many cases, non-English publications were the predominant or only literature available, resulting in a lack of analysable data after their exclusion.

Conclusions

In general, studies published in non-English languages have little influence on the conclusions of HTA reports. For the vast majority of topics, a language restriction to English seems justified. Studies published in non-English languages may be useful in exceptional cases, for example when an intervention is only available in certain countries.

Keywords

Language; Publication Bias; Publications; Retrospective Studies

Corresponding author: Elke Hausner

Competing interests: No competing interests were disclosed.

Grant information: The author(s) declared that no grants were involved in supporting this work.

Copyright: © 2024 Hausner E et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Hausner E, Sturtz S, Molnar S et al. Searching for non-English literature may be unnecessary for HTA Reports [version 1; peer review: 2 approved with reservations, 1 not approved]. F1000Research 2024, 13:1134 (https://doi.org/10.12688/f1000research.151365.1) First published: 04 Oct 2024, 13:1134 (https://doi.org/10.12688/f1000research.151365.1) Latest published: 07 Jul 2025, 13:1134 (https://doi.org/10.12688/f1000research.151365.3)

Introduction

Health technology assessments (HTAs), which usually include systematic reviews, are based on comprehensive information retrieval requiring the use of multiple information sources.¹^,² Current standards discourage the use of search restrictions, such as publication date and language.¹^,³ This is justified by the assumption that otherwise HTA conclusions could be biased, as studies with statistically significant results are more likely to be published in English-language journals, whereas non-significant results are more likely to be published in journals in languages other than English.⁴ However, there is now evidence that significant results are increasingly being published in non-English journals.⁵

In practice, many systematic reviewers restrict their searches to English-language articles.⁶^,⁷ The reasons for this are pragmatic: non-English literature is often more difficult to obtain and translation costs are high.⁸

The Institute for Quality and Efficiency in Health Care (IQWiG⁹) is a German HTA agency. In addition to more than 100 HTAs based on dossiers submitted by drug companies, IQWiG conducts around 10-15 other HTAs per year, mostly on non-drug interventions. The HTA reports investigate patient-relevant endpoints or validated surrogate endpoints and may comprise a number of different PICO questions on population, intervention, comparator, and outcomes. If possible, results of single studies are pooled in meta-analyses. The overall certainty of conclusions is graded into 3 levels of increasing certainty (hint, indication or proof of harm or benefit) according to the amount and quality of the available evidence. A detailed overview of IQWiG’s methods is provided in its methods paper.¹⁰

IQWiG’s reports are generally based on comprehensive information retrieval. However, it was previously unclear whether it was worthwhile searching for, screening, and translating non-English publications on primary studies, i.e. whether their inclusion influenced the conclusions of the reports.

Aim

The aim of this analysis was to assess the impact of non-English publications on the conclusions of HTA reports.

Methods

Following an internal protocol, we screened HTA reports (all reports on non-drug interventions published by IQWiG between 06/2007 and 08/2018 and randomly selected reports on drugs published between 09/2011 and 11/2016) for the inclusion of non-English publications. This procedure was not updated to include more recent HTA reports in the present article, as from September 2018 onwards, IQWiG restricted its searches to publications with English or German full texts. We focused on non-drug interventions, as we expected a higher number of non-English publications in this area.

In order to identify non-English publications, we screened all publications listed in the study pools of the HTA reports. This step was carried out via the EndNote databases of the underlying projects or, for older projects, via the reference lists in the HTA reports. In the next step, the language of the publications was identified using the entries in the bibliographic databases (MEDLINE, Embase). In a few cases this was not possible, so the language was checked using the full text or journal description. Data were extracted by one person and checked by another.

Only journal publications were included in the analysis, i.e., unpublished reports, conferences abstracts, evidence syntheses, clinical study reports, or registry entries were not considered. For updates of HTA reports, only the newly identified references were taken into account. For all HTA reports that included at least one non-English publication, we analysed whether the exclusion of such publications led to a change in conclusions; this was done separately for each PICO question and, if applicable, for each endpoint using specific categories (Table 1).

Table 1. Classification of non-English publication.

Category number

Classification of non-English publication

Change in the conclusions of the HTA report

Conclusion does not change

• Non-English study with low weight for PICO
• All studies including non-English publication point into the same direction
• Higher level of evidence compared to non-English publication available
• Amount and quality of the evidence of all studies included in the HTA report was too low for reliable conclusions, regardless of inclusion of non-English publication
• All studies included have unclear clinical relevance
• Diagnostic study without direct relevance for the HTA report

Publication without relevance for the HTA report

• Quality of study too low to change conclusion
• A secondary publication in English was included
• No data from non-English publication was used in the HTA report
• Non-English publication was not translated

Analysis

Using standardized methods (IQWiG’s methods paper¹⁰ and internal guidance), we re-assessed the conclusions for the affected endpoints and PICO questions in the HTA reports. We did not re-analyse HTA reports on diagnostic test accuracy (DTA), as no conclusion on test accuracy endpoints is drawn in such reports.

For this purpose, the project manager reviewed all reports and assessed the impact of the exclusion of the non-English publication(s). This included the examination of individual studies on a specific endpoint or PICO question, qualitative re-assessments, and, in 3 cases, meta-analyses that had to be recalculated. When in doubt, the project manager consulted with another project manager or statistician.

For each endpoint, we defined a change in a conclusion due to the exclusion of non-English publications as either an upgrading or downgrading of the certainty of the conclusion or a complete loss of data (no conclusion possible). The effect of the exclusion of non-English publications on conclusions was categorized for each HTA report as follows: Category A: change in the conclusion for a particular endpoint; Category B: no change in the conclusion for a particular endpoint; Category C: the non-English publications do not contain relevant information for the HTA report (e.g. because the amount or quality of the data is too low) and are therefore classified as “not relevant”. The categorization was carried out independently by the project manager and a statistician. Any discrepancies were resolved by consensus between the two.

Results

70 eligible HTA reports including 2339 publications were identified³⁰ (Figure 1). Due to the inclusion of only a small number of selected drug reports, the vast majority of the HTA reports (96%) were on non-drug interventions (Table 2).

Figure 1. Flowchart on the presence and impact of non-English publications.

Table 2. Characteristics of the HTA reports.

Characteristics of the HTA reports	Number (percentage) or median [minimum; maximum]
Type of intervention • Drug • Non-drug ○ Diagnostic ○ Screening ○ Treatment	3 (4) 67 (96) 16 (23) 16 (23) 35 (50)
Number of publications included • Number per report	2339 22.5 [1; 156]
Design of studies included (defined a priori) • RCTs only • RCTs + other study designs	33 (47) 37 (53)
Number of bibliographic databases^a	3 [3; 23]

a In addition, information retrieval for the HTA reports usually included a search of study registries and a check of reference lists of systematic reviews. Furthermore, requests to authors were made, if necessary, and public hearings took place (except for rapid reports).

Of the 70 HTA reports, 32 (46%) included English-language publications only (Table 3). The remaining 38 reports (54%) included 128 non-English publications. These were most commonly published in German, Chinese, French and Spanish (Table 4).

Table 3. Number of non-English publications included per HTA report.

Number of non-English publications included	Number of HTA reports
0	32
1	16
2	4
3-11	18

Table 4. Languages of non-English publications included in the HTA reports.

Language	Number (%)^a
German	43 (1.8%)
Chinese	30 (1.3%)
French	12 (0.5%)
Spanish	10 (0.4%)
other	33 (1.4%)

a % of all included 2339 publications.

HTA reports with a change in the conclusions (Category A)

The 4 reports with a change in the conclusions were (short titles):

• A05-18: Tiotropium bromide for chronic obstructive pulmonary disease,¹¹^,¹²
• N14-02: Systemic psychotherapy in adults¹³^,¹⁴
• N16-01: Active knee motion devices for anterior cruciate ligament ruptures¹⁵^,¹⁶
• N16-03: Continuous passive motion (CPM) devices after knee or shoulder surgery¹⁷^,¹⁸

For further details, see Table 5.

Table 5. HTA reports with endpoints that resulted in a change in the conclusions (Category A).

HTA reports	Title	Total number of PICO questions	Conclusion changed (PICO)	Studies published in non-English languages where exclusion from HTA report changed conclusions	Evaluated EP per affected PICO question	Conclusions changed (EP)
A05-18	Tiotropium bromide in the treatment of chronic obstructive pulmonary disease	12	2	Fang 2008 (Chi), Jia 2008 (Chi)	2	2
A05-18		12	2	Fang 2008 (Chi), Jia 2008 (Chi)	9	1
N14-02	Systemic therapy in adults as a psychotherapeutic approach	18	4	Li 2010 (Chi), Yang 2005 (Chi), Wang 2011 (Chi), Wirsching 1989 (Ger), Cao 2007 (Chi), Zhang 2006^a (Chi)	2	2
					6	1
					17	1
					9	2
N16-01	Active knee motion devices in the treatment of anterior cruciate ligament ruptures	2	1	Von Lübken 2006 (Ger)	2	2
N16-03	Motor-driven continuous passive motion (CPM) devices after interventions on the knee and shoulder joint	7	1	Michael 2005 (Ger)	3	2

a 2 publications are available for the Zhang 2006 study: Zhang 2006a and Zhang 2006b.

These 4 reports investigated 50 endpoints in 39 PICO questions. In 3 out of the 4 reports (A05-18, N14-02, N16-01), the only available publication on a particular endpoint was a non-English publication. The exclusion of the non-English publication led to a change in the conclusion for 13 endpoints in 8 PICO questions (Table 5). For 5 of these endpoints, the conclusion changed from a non-significant effect or an inconclusive result to “no conclusion possible” (due to a lack of data), and for 7 endpoints with previously statistically significant results, no conclusions could be drawn due to lack of data. For 1 endpoint, the conclusion changed from an inconclusive result to a hint of a benefit of the test intervention due to an effect becoming statistically significant (see example below).

Of the 10 studies that influenced the conclusions, 7 were published in Chinese and 3 in German (Table 4).

Example from an HTA report

Report N16-03 investigated the use of continuous passive motion (CPM) devices after knee or shoulder surgery. For the comparison of shoulder devices in combination with physical therapy versus physical therapy alone in patients with rotator cuff rupture, there were 2 studies on the endpoint “pain”, one in English (Garofalo 2010) and one in German (Michael 2005). When both studies were considered, the results for this endpoint were inconsistent (see Figure 2), and therefore no conclusion could be drawn. When Michael 2005 was excluded, there was a hint of a benefit for combination therapy, i.e. the exclusion of non-English literature changed the conclusion.

Figure 2. Report N16-03: Forest plot for the endpoint “pain”.

HTA reports with no change in the conclusions (Category B)

If the exclusion of non-English publications did not change the conclusions of the HTA report, we identified the reason for this for each endpoint. In most cases, this was because there were several studies for each endpoint with results pointing in the same direction, meaning that the exclusion of a non-English publication had a negligible effect.

Example from an HTA report

Report N09-01 investigated different non-drug local treatments in patients with benign prostatic syndrome. For the comparison of holmium laser therapy (HoLEP) versus standard treatment, 6 studies were included for the endpoint “symptom scores at 3 months”, of which one was in Chinese (Zhang 2007). Neither the result of a single study nor the pooled effect estimate showed a statistically significant effect. Zhang 2007 contributed a weight of 16.1% to the pooled effect estimate (see Figure 3); its exclusion did not change the conclusion.

Figure 3. Report N09-01: Forest plot for the endpoint “symptom scores at 3 months”.

Non-English publications classified as not relevant (Category C)

For the non-English publications assigned to Category C, the studies formally met the inclusion criteria of the HTA reports, and were therefore included in the study pool, but were not used in the actual assessment, mainly due to low data quality.

Discussion

Our analysis shows that the exclusion of studies published in non-English languages has only a minor effect on the conclusions of HTA reports. This is consistent with the results of a recent systematic review by Dobrescu et al. on the restriction of systematic reviews of diagnostic or treatment interventions to English-language publications. They found that the inclusion of solely English-language publications led to a change in statistical significance in only 23 out of 259 meta-analyses (9%) and concluded that “the impact of restricting systematic reviews to English-language publications is negligible for most conventional medicine topics”.¹⁹ Dobrescu et al.’s article was primarily based on the analysis of Cochrane reviews. Our analysis shows that the results can also be transferred to the context of HTA reports. In our analysis, most studies with an impact on the conclusions of the HTA reports were published in Chinese. This is notable, as the volume of Chinese-language literature is growing rapidly and there have been calls for the inclusion of such publications.²⁰^,²¹ However, their quality has been questioned because of methodological flaws and overly positive results.⁵

On the basis of our analysis, it is not possible to identify topics where non-English publications might be relevant. However, it seems reasonable to conduct additional searches for non-English literature if a particular intervention is exclusively or preferentially available in a specific region. For example, the “continuous active motion device” is a German product that is hardly marketed outside Germany. Only 2 studies could be identified for the HTA; both were published by German authors, one in English and one in German.

Language restrictions can save time. If an HTA report is being prepared under considerable time pressure, the exclusion of non-English literature should already take place at the level of the search strategy, resulting in a lower number of hits to be screened.

Limitations

Methods for meta-analysis and for the assessment of study quality have changed over the past decades, which may influence the conclusions of HTA reports. We therefore tried to follow the original methods of the HTA reports included.

Only a small proportion of HTA reports required a recalculation of results.

At IQWiG, the analysis presented here led to the restriction of information retrieval in HTA reports to German- and English-language literature from September 2018 onwards. This change in approach is also described in IQWiG’s updated methods paper.¹⁰ For the present article, it was therefore not meaningful to update the pool of HTA reports by including HTA reports published by IQWiG after October 2019. An evaluation of current HTA reports from other large HTA agencies shows that they generally use a language restriction in their reports.²²^–²⁹

We did not analyse whether the non-English study publications were of lower quality than the English-language ones.

For 2 reports (D06-01B and D06-01C), we also included evidence syntheses, although this document type was to be excluded. One report (D06-01M) was included although only a preliminary version was available. As the results of these 3 reports did not influence the analysis, they were not subsequently removed from the database. Because only a small number of selected drug reports were included, the vast majority of the HTA reports were on non-drug interventions. Therefore, no conclusive statement can be made about drug reports.

Conclusions

Ethics and consent

Ethics and consent are not required.

Data availability

Underlying data

Zenodo: Searching for non-English literature may be unnecessary for HTA Reports - supplemental material [Data set]. https://zenodo.org/records/13340928.

▪ Supplementary-material_matrix_V2.xlsx (Data set of extracted references with details of the publication language)³⁰

Data are available under the terms of the Creative Commons Attribution 4.0 International License (CC-BY 4.0).

Extended data

Zenodo: Searching for non-English literature may be unnecessary for HTA Reports - supplemental material [Data set].https://doi.org/10.5281/zenodo.12642960.

▪ Supplementary-material_table.docx (HTA reports with endpoints that resulted in a change in the conclusion (Category A and B)³¹

Data are available under the terms of the Creative Commons Attribution 4.0 International License (CC-BY 4.0).

Acknowledgments

We thank Anke Schulz (calculations), Wolfram Groß (data curation), and Natalie McGauran (translation and editing).

References

1. Lefebvre C, Glanville J, Briscoe S, et al.: Chapter 4: Searching for and selecting studies.2023. [Accessed: 07.02.2024]. Reference Source
2. EUnetHTA JA3WP6B2-2 Authoring Team: Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness. Methodological Guidelines.2019. [Accessed: 06.02.2024]. Reference Source
3. Relevo R, Balshem H: Finding evidence for comparing medical interventions: AHRQ and the Effective Health Care Program. J. Clin. Epidemiol. 2011; 64(11): 1168–1177. PubMed Abstract | Publisher Full Text
4. Egger M, Zellweger-Zahner T, Schneider M, et al.: Language bias in randomised controlled trials published in English and German. Lancet. 1997; 350(9074): 326–329. Publisher Full Text
5. Wu T, Li Y, Bian Z, et al.: Randomized trials published in some Chinese journals: how many are randomized? Trials. 2009; 10: 46. PubMed Abstract | Publisher Full Text | Free Full Text
6. Jackson JL, Kuriyama A: How Often Do Systematic Reviews Exclude Articles Not Published in English? J. Gen. Intern. Med. 2019; 34(8): 1388–1389. PubMed Abstract | Publisher Full Text | Free Full Text
7. Hoffmann F, Allers K, Rombey T, et al.: Nearly 80 systematic reviews were published each day: Observational study on trends in epidemiology and reporting over the years 2000-2019. J. Clin. Epidemiol. 2021; 138: 1–11. PubMed Abstract | Publisher Full Text
8. Morrison A, Moulton K, Clark M, et al.: English-Language Restriction When Conducting Systematic Review-based Meta-analyses: Systematic Review of Published Studies.2009. [Accessed: 06.02.2024]. Reference Source
9. Institute for Quality and Efficiency for in Health Care: IQWiG.2024. [Accessed: 08.01.2024]. Reference Source
10. Institute for Quality and Efficiency for in Health Care: General Methods. [Accessed: 08.01.2024]. Reference Source
11. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen: Tiotropiumbromid bei COPD; Abschlussbericht.2012. [Accessed: 06.02.2024]. Reference Source
12. Institute for Quality and Efficiency in Health Care: Tiotropium bromide for COPD; Executive Summary.2012. [Accessed: 06.02.2024]. Reference Source
13. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen: Systemische Therapie bei Erwachsenen als Psychotherapieverfahren; Abschlussbericht.2017. [Accessed: 06.02.2024]. Reference Source
14. Institute for Quality and Efficiency in Health Care: Systemic therapy in adults as a psychotherapeutic approach; Extract.2017. [Accessed: 06.02.2024]. Reference Source
15. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen: Aktive Kniebewegungsschienen in der Behandlung von Rupturen des vorderen Kreuzbands; Abschlussbericht.2017. [Accessed: 06.02.2024]. Reference Source
16. Institute for Quality and Efficiency in Health Care: Active knee motion devices in the treatment of anterior cruciate ligament ruptures; Extract.2017. [Accessed: 06.02.2024]. Reference Source
17. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen: Einsatz von motorbetriebenen Bewegungsschienen (CPM) nach Interventionen am Knie- oder am Schultergelenk; Abschlussbericht.2018. [Accessed: 06.02.2024]. Reference Source
18. Institute for Quality and Efficiency for in Health Care: Motor-driven continuous passive motion (CPM) devices after interventions in the knee and shoulder joint; Extract.2018. [Accessed: 06.02.2024]. Reference Source
19. Dobrescu AI, Nussbaumer-Streit B, Klerings I, et al.: Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review. J. Clin. Epidemiol. 2021; 137: 209–217. PubMed Abstract | Publisher Full Text
20. Xue J, Chen W, Chen L, et al.: Significant discrepancies were found in pooled estimates of searching with Chinese indexes versus searching with English indexes. J. Clin. Epidemiol. 2016; 70: 246–253. PubMed Abstract | Publisher Full Text
21. Cohen JF, Korevaar DA, Wang J, et al.: Should we search Chinese biomedical databases when performing systematic reviews? Syst. Rev. 2015; 4(1): 23. PubMed Abstract | Publisher Full Text | Free Full Text
22. Wilson M, Wang Z, Dobler CC, et al.: Noninvasive Positive Pressure Ventilation in the Home (with addendum).2020. [Accessed: 06.02.2024]. Reference Source
23. National Institute for Health and Care Excellence: Remsima (infliximab biosimilar) for subcutaneous injection for managing Crohn’s disease and ulcerative colitis; Evidence review.2021. [Accessed: 06.02.2024]. Reference Source
24. National Institute for Health and Care Excellence: Antimicrobial prescribing: delafloxacin for community-acquired pneumonia; Evidence review.2021. [Accessed: 06.02.2024]. Reference Source
25. Haute Autorité de santé: Suivi par télésurveillance des patients porteurs d’un moniteur cardiaque implantable.2021. [Accessed: 06.02.2024]. Reference Source
26. Haute Autorité de santé: Evaluation du parage de plaie de la pulpe par coiffage pulpaire direct.2019. [Accessed: 06.02.2024]. Reference Source
27. Farrell K, Horton J: Airway Pressure Release Ventilation for Acute Respiratory Distress Syndrome: An Update. Can. J. Health Technol. 2022; 2(9). Publisher Full Text
28. Chou R, Cantor A, Dana T, et al.: Statin Use for the Primary Prevention of Cardiovascular Disease in Adults: A Systematic Review for the U.S. Preventive Services Task Force.2022. [Accessed: 06.02.2024]. Reference Source
29. Barbara AM, Grobelna A: Real-Time Continuous Glucose Monitoring for People Living With Type 1 Diabetes. Can. J. Health Technol. 2022; 2(8). Publisher Full Text
30. Hausner E: Searching for non-English literature may be unnecessary for HTA Reports - supplemental material. Supplementary-material_matrix_V2. [Data set]. Zenodo. 2024. Reference Source
31. Hausner E: Searching for non-English literature may be unnecessary for HTA Reports - supplemental material. Supplementary-material_table. [Data set]. Zenodo. 2024. Publisher Full Text

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 04 Oct 2024

Author details Author details

Elke Hausner
Roles: Conceptualization, Methodology, Project Administration, Writing – Original Draft Preparation

Sibylle Sturtz
Roles: Data Curation, Formal Analysis, Methodology, Writing – Original Draft Preparation

Sandra Molnar
Roles: Data Curation, Methodology, Writing – Original Draft Preparation

Lisa Schell
Roles: Conceptualization, Data Curation, Methodology, Project Administration, Writing – Original Draft Preparation

Wiebke Sieben
Roles: Data Curation, Formal Analysis, Methodology, Writing – Review & Editing

Stefan Sauerland
Roles: Conceptualization, Supervision, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Article Versions (3)

version 3

Revised

Published: 07 Jul 2025, 13:1134

https://doi.org/10.12688/f1000research.151365.3

version 2

Revised

Published: 03 Apr 2025, 13:1134

https://doi.org/10.12688/f1000research.151365.2

version 1

Published: 04 Oct 2024, 13:1134

https://doi.org/10.12688/f1000research.151365.1

© 2024 Hausner E et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Hausner E, Sturtz S, Molnar S et al. Searching for non-English literature may be unnecessary for HTA Reports [version 1; peer review: 2 approved with reservations, 1 not approved]. F1000Research 2024, 13:1134 (https://doi.org/10.12688/f1000research.151365.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 1

VERSION 1

PUBLISHED 04 Oct 2024

Views

Reviewer Report 28 Dec 2024

Tarquin Mittermayr, Austrian Institute for Health Technology Assessments, Wien, Austria

Approved with Reservations

https://doi.org/10.5256/f1000research.166008.r344992

I am happy to be invited to write a review on this relevant topic.
Whilst the method for assessing the impact of excluding non-english literature on the conclusion of HTA reports is generally explained in a clear and comprehensible manner, Hausner et al fall short of explaining or referencing the internal protocol mentioned in the methods section.

It would have also been helpful to either provide an explanation already in this section or refer to the appropriate page (e.g. “please find details on page 8”) in the limitations section as to why no publications after 2018 were considered for this assessment.

Table 1 would benefit from clearer and more concise information:
The same reasons for classifying low quality studies as category B in the classification table (“Amount and quality of the evidence…”) are listed in the explanation of the categories below the table under category C, thus giving room for ambiguous interpretations.

Table 2 contains the total number of HTA reports included as well as the number per report. It is unclear if the number given for the latter (22.5) is an average, median or a percentage. In consideration of whether providing this number is meaningful, stating just the range of included publications per report might suffice and be more explanatory.

The discussion section contains the confusing remark “We did not analyse whether the non-English study publications were of lower quality than the English language ones”. This appears to be contradictory to the explanations given for the classifications in Table 1. Should the contradictory nature of this statement be due to an imprecision in terminology (quality of evidence vs. quality of study publications), this study by Hausner et al would have benefited from performing a qualitative analyses of the publications themselves (in addition to analysing the containing evidence).

In an article which argues that excluding non-English literature may have little impact on the overall conclusion of an HTA report, the example from Report N16-03 given in the Results section, where the exclusion of a non-English article led to a change in the conclusion, might sit uncomfortably with systematic evidence based medicine professionals. As graphical this example may be in showing the effect of this exclusion, it also appears to undermine the core argument of this article.

A revision by the authors of this publication should focus on the coherence of their argumentation on the one hand, and on supporting their methods with more concise examples (e.g. Tables and Figures).

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

I cannot comment. A qualified statistician is required.
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Conducting systematic searches in my position as an Information Specialist

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 03 Apr 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

03 Apr 2025

Author Response

Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The reviewer indicates that it would ... Continue reading Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The reviewer indicates that it would be helpful to explain or reference the internal protocol mentioned in the methods section.

Reply: We have exchanged the term “protocol” with “project outline”. This document is a German-language 1-page text that provides an overview of the project, not a complete step-by-step protocol. It was not published and we therefore cannot reference it.

Change in the text: We have amended the text
“Following an internal project outline, we screened HTA reports (all reports on non-drug interventions published by IQWiG between 06/2007 and 08/2018 and randomly selected reports on drugs published between 09/2011 and 11/2016) for the inclusion of non-English publications.”

2. The reviewer states that it would have been helpful to either provide an explanation already in this section or refer to the appropriate page (e.g. “please find details on page 8”) in the limitations section as to why no publications after 2018 were considered for this assessment.

Change in the text: We have amended the text

”For the present article, it was therefore not meaningful to update the pool of HTA reports by including HTA reports published by IQWiG after October 2019 (see the methods section for details).“

3. The reviewer states that table 1 would benefit from clearer and more concise information: The same reasons for classifying low quality studies as category B in the classification table (“Amount and quality of the evidence…”) are listed in the explanation of the categories below the table under category C, thus giving room for ambiguous interpretations.

Reply: We disagree that there is ambiguity, because in category B the reason is “Amount and quality of the evidence of all studies”, whereas in category C the reason is “Quality of study too low to change conclusion”. The first reason refers to the evidence in its totality, the second reason addresses the quality of only the non-English study.

No change to the text

4. The reviewer points out that in Table 2 it is unclear whether the number of included publications per report (22.5) is an average, median or percentage.

Reply: Numbers and percentages are indicated by round brackets, while median and ranges are reported with square brackets. The heading indicates that 22.5 [1; 156] is the median, with the range given. We believe that it is reasonable to display median and range, rather than just reporting the range.

No change to the text

5. The reviewer refers to the different information from Table 1 and the section ‘Limitations’ regarding the examination of study quality. In addition, the reviewer states that the study would have benefited from performing a qualitative analysis for publications.

Reply: We are uncertain what type of “qualitative analysis” could be useful to be performed in addition to analysing the available evidence. As HTA work is evidence-based, the methods and the results of the primary studies are the sole data sources. Unfortunately, a qualitative analysis was not possible due to the limited time available. However, this would be very interesting, as there are already several publications [1-3] on this topic that question the quality of non-English studies.

Change in the text: We have added the assessment of the risk of bias for the non-English studies in the text.

“However, all 10 non-English studies that changed the conclusions of an HTA report (see Table 5) were classified as highly biased in the underlying HTA reports.”

6. The reviewer points out that the example of N16-03 listed under ‘Results’ could lead to discomfort among experts in evidence-based medicine since it seems to undermine the core argument of this article.

Reply: Admittedly, the recommendation not to search for non-English literature would have been easier if the decision to include or exclude such sources had not affected the conclusion of any HTA report. As 4 reports (6%) experienced a change in the conclusions, our conclusion is already somewhat toned down (“seems justified”). We agree with the reviewer that the decision is not straightforward. Some experts will continue to search non-English sources, because they consider the risk of overlooking a study more important than the resources required for searching for non-English publications. In our view, however, the main conclusion is correctly inferred from our results.

No change to the text
Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The reviewer indicates that it would be helpful to explain or reference the internal protocol mentioned in the methods section.

Reply: We have exchanged the term “protocol” with “project outline”. This document is a German-language 1-page text that provides an overview of the project, not a complete step-by-step protocol. It was not published and we therefore cannot reference it.

Change in the text: We have amended the text
“Following an internal project outline, we screened HTA reports (all reports on non-drug interventions published by IQWiG between 06/2007 and 08/2018 and randomly selected reports on drugs published between 09/2011 and 11/2016) for the inclusion of non-English publications.”

2. The reviewer states that it would have been helpful to either provide an explanation already in this section or refer to the appropriate page (e.g. “please find details on page 8”) in the limitations section as to why no publications after 2018 were considered for this assessment.

Change in the text: We have amended the text

”For the present article, it was therefore not meaningful to update the pool of HTA reports by including HTA reports published by IQWiG after October 2019 (see the methods section for details).“

3. The reviewer states that table 1 would benefit from clearer and more concise information: The same reasons for classifying low quality studies as category B in the classification table (“Amount and quality of the evidence…”) are listed in the explanation of the categories below the table under category C, thus giving room for ambiguous interpretations.

Reply: We disagree that there is ambiguity, because in category B the reason is “Amount and quality of the evidence of all studies”, whereas in category C the reason is “Quality of study too low to change conclusion”. The first reason refers to the evidence in its totality, the second reason addresses the quality of only the non-English study.

No change to the text

4. The reviewer points out that in Table 2 it is unclear whether the number of included publications per report (22.5) is an average, median or percentage.

Reply: Numbers and percentages are indicated by round brackets, while median and ranges are reported with square brackets. The heading indicates that 22.5 [1; 156] is the median, with the range given. We believe that it is reasonable to display median and range, rather than just reporting the range.

No change to the text

5. The reviewer refers to the different information from Table 1 and the section ‘Limitations’ regarding the examination of study quality. In addition, the reviewer states that the study would have benefited from performing a qualitative analysis for publications.

Reply: We are uncertain what type of “qualitative analysis” could be useful to be performed in addition to analysing the available evidence. As HTA work is evidence-based, the methods and the results of the primary studies are the sole data sources. Unfortunately, a qualitative analysis was not possible due to the limited time available. However, this would be very interesting, as there are already several publications [1-3] on this topic that question the quality of non-English studies.

Change in the text: We have added the assessment of the risk of bias for the non-English studies in the text.

“However, all 10 non-English studies that changed the conclusions of an HTA report (see Table 5) were classified as highly biased in the underlying HTA reports.”

6. The reviewer points out that the example of N16-03 listed under ‘Results’ could lead to discomfort among experts in evidence-based medicine since it seems to undermine the core argument of this article.

Reply: Admittedly, the recommendation not to search for non-English literature would have been easier if the decision to include or exclude such sources had not affected the conclusion of any HTA report. As 4 reports (6%) experienced a change in the conclusions, our conclusion is already somewhat toned down (“seems justified”). We agree with the reviewer that the decision is not straightforward. Some experts will continue to search non-English sources, because they consider the risk of overlooking a study more important than the resources required for searching for non-English publications. In our view, however, the main conclusion is correctly inferred from our results.

No change to the text
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 03 Apr 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

03 Apr 2025

Author Response

Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The reviewer indicates that it would ... Continue reading Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The reviewer indicates that it would be helpful to explain or reference the internal protocol mentioned in the methods section.

Reply: We have exchanged the term “protocol” with “project outline”. This document is a German-language 1-page text that provides an overview of the project, not a complete step-by-step protocol. It was not published and we therefore cannot reference it.

Change in the text: We have amended the text
“Following an internal project outline, we screened HTA reports (all reports on non-drug interventions published by IQWiG between 06/2007 and 08/2018 and randomly selected reports on drugs published between 09/2011 and 11/2016) for the inclusion of non-English publications.”

2. The reviewer states that it would have been helpful to either provide an explanation already in this section or refer to the appropriate page (e.g. “please find details on page 8”) in the limitations section as to why no publications after 2018 were considered for this assessment.

Change in the text: We have amended the text

”For the present article, it was therefore not meaningful to update the pool of HTA reports by including HTA reports published by IQWiG after October 2019 (see the methods section for details).“

3. The reviewer states that table 1 would benefit from clearer and more concise information: The same reasons for classifying low quality studies as category B in the classification table (“Amount and quality of the evidence…”) are listed in the explanation of the categories below the table under category C, thus giving room for ambiguous interpretations.

Reply: We disagree that there is ambiguity, because in category B the reason is “Amount and quality of the evidence of all studies”, whereas in category C the reason is “Quality of study too low to change conclusion”. The first reason refers to the evidence in its totality, the second reason addresses the quality of only the non-English study.

No change to the text

4. The reviewer points out that in Table 2 it is unclear whether the number of included publications per report (22.5) is an average, median or percentage.

Reply: Numbers and percentages are indicated by round brackets, while median and ranges are reported with square brackets. The heading indicates that 22.5 [1; 156] is the median, with the range given. We believe that it is reasonable to display median and range, rather than just reporting the range.

No change to the text

5. The reviewer refers to the different information from Table 1 and the section ‘Limitations’ regarding the examination of study quality. In addition, the reviewer states that the study would have benefited from performing a qualitative analysis for publications.

Reply: We are uncertain what type of “qualitative analysis” could be useful to be performed in addition to analysing the available evidence. As HTA work is evidence-based, the methods and the results of the primary studies are the sole data sources. Unfortunately, a qualitative analysis was not possible due to the limited time available. However, this would be very interesting, as there are already several publications [1-3] on this topic that question the quality of non-English studies.

Change in the text: We have added the assessment of the risk of bias for the non-English studies in the text.

“However, all 10 non-English studies that changed the conclusions of an HTA report (see Table 5) were classified as highly biased in the underlying HTA reports.”

6. The reviewer points out that the example of N16-03 listed under ‘Results’ could lead to discomfort among experts in evidence-based medicine since it seems to undermine the core argument of this article.

Reply: Admittedly, the recommendation not to search for non-English literature would have been easier if the decision to include or exclude such sources had not affected the conclusion of any HTA report. As 4 reports (6%) experienced a change in the conclusions, our conclusion is already somewhat toned down (“seems justified”). We agree with the reviewer that the decision is not straightforward. Some experts will continue to search non-English sources, because they consider the risk of overlooking a study more important than the resources required for searching for non-English publications. In our view, however, the main conclusion is correctly inferred from our results.

No change to the text
Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The reviewer indicates that it would be helpful to explain or reference the internal protocol mentioned in the methods section.

Reply: We have exchanged the term “protocol” with “project outline”. This document is a German-language 1-page text that provides an overview of the project, not a complete step-by-step protocol. It was not published and we therefore cannot reference it.

Change in the text: We have amended the text
“Following an internal project outline, we screened HTA reports (all reports on non-drug interventions published by IQWiG between 06/2007 and 08/2018 and randomly selected reports on drugs published between 09/2011 and 11/2016) for the inclusion of non-English publications.”

2. The reviewer states that it would have been helpful to either provide an explanation already in this section or refer to the appropriate page (e.g. “please find details on page 8”) in the limitations section as to why no publications after 2018 were considered for this assessment.

Change in the text: We have amended the text

”For the present article, it was therefore not meaningful to update the pool of HTA reports by including HTA reports published by IQWiG after October 2019 (see the methods section for details).“

3. The reviewer states that table 1 would benefit from clearer and more concise information: The same reasons for classifying low quality studies as category B in the classification table (“Amount and quality of the evidence…”) are listed in the explanation of the categories below the table under category C, thus giving room for ambiguous interpretations.

Reply: We disagree that there is ambiguity, because in category B the reason is “Amount and quality of the evidence of all studies”, whereas in category C the reason is “Quality of study too low to change conclusion”. The first reason refers to the evidence in its totality, the second reason addresses the quality of only the non-English study.

No change to the text

4. The reviewer points out that in Table 2 it is unclear whether the number of included publications per report (22.5) is an average, median or percentage.

Reply: Numbers and percentages are indicated by round brackets, while median and ranges are reported with square brackets. The heading indicates that 22.5 [1; 156] is the median, with the range given. We believe that it is reasonable to display median and range, rather than just reporting the range.

No change to the text

5. The reviewer refers to the different information from Table 1 and the section ‘Limitations’ regarding the examination of study quality. In addition, the reviewer states that the study would have benefited from performing a qualitative analysis for publications.

Reply: We are uncertain what type of “qualitative analysis” could be useful to be performed in addition to analysing the available evidence. As HTA work is evidence-based, the methods and the results of the primary studies are the sole data sources. Unfortunately, a qualitative analysis was not possible due to the limited time available. However, this would be very interesting, as there are already several publications [1-3] on this topic that question the quality of non-English studies.

Change in the text: We have added the assessment of the risk of bias for the non-English studies in the text.

“However, all 10 non-English studies that changed the conclusions of an HTA report (see Table 5) were classified as highly biased in the underlying HTA reports.”

6. The reviewer points out that the example of N16-03 listed under ‘Results’ could lead to discomfort among experts in evidence-based medicine since it seems to undermine the core argument of this article.

Reply: Admittedly, the recommendation not to search for non-English literature would have been easier if the decision to include or exclude such sources had not affected the conclusion of any HTA report. As 4 reports (6%) experienced a change in the conclusions, our conclusion is already somewhat toned down (“seems justified”). We agree with the reviewer that the decision is not straightforward. Some experts will continue to search non-English sources, because they consider the risk of overlooking a study more important than the resources required for searching for non-English publications. In our view, however, the main conclusion is correctly inferred from our results.

No change to the text
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 14 Dec 2024

Sonia Garcia Gonzalez-Moral, Newcastle University, Newcastle upon Tyne, England, UK

Approved with Reservations

https://doi.org/10.5256/f1000research.166008.r344994

I welcome the opportunity to comment on this paper. With this study the authors have challenged the status quo of current systematic literature review guidelines such as those from Cochrane and health technology assessment methods recommendations for undertaking literature searches such as those recommended by EUneHTA. Such bold move feels justified if one understand the intensity of resource and capacity involved in HTA and so, while I support the rationale for this study and think that this is a good first attempt to breach this topic, in general I do not think that this study boasts the sufficient methodological rigor nor a big enough sample size to draw conclusions that could be generalized to HTA.
To start with, the title of this manuscript ‘Searching for non-English literature may be unnecessary for HTA reports’ suggests the opposite of what the authors conclude ‘At IQWiG, the analysis presented here led to the restriction of information retrieval in HTA reports to German- and English-language literature from September 2018 onwards’. I would therefore recommend that the title is revised, contextualizing this study to the needs and experience of the German HTA agency may increase the accuracy of their findings and conclusions.
In the Introduction section the authors draw from one single study (Wu et al., 2009) to justify why non-English studies should no longer be searched for in HTA reports. Wu et al. analysed how randomised controlled trials were being reported in the Chinese literature written in Chinese. Their paper seemed to suggest that significant results were being published in languages other than English (in their case Chinese) and that RCTs were often poorly reported in Chinese. However, the authors of this manuscript later in the Discussion counterargue this point by remarking that ‘[…] as the volume of Chinese-language literature is growing rapidly and there have been calls for the inclusion of such publications. However, their quality has been questioned because of methodological flaws and overly positive results.’ Deviating from the point of publication bias and focusing on the quality of the research as a reason for not searching non- English language publications in HTA. While the quality of research is extremely important, there are instruments that deal with that issue within the review process making ‘poor quality’ (according to Wu’s study) not a priory reason for not searching the entire scientific literature written on a particular language such as Chinese. In light of this contradiction the rationale for this study should be reformulated to accurately justify the reasons for not searching non-English papers and always placing these in the context of the German HTA agency. So far, the reasons presented here do not seem to warrant such claim.
There are a number of omissions and inaccuracies through the methods section such as the search period in the abstract is not in line with the period reported in the Methods rendering this point unclear. Further, having read the full paper twice, I am still unsure of whether HTA reports of drugs were or not included in their assessment. I think the authors should provide a detailed list of the HTA reports (medicines and non-medicines) that were screened for inclusion in the methods section since these are not their results but the body of data they used for analysis. They should also justify why diagnostic HTA reports would have been included from the start if they were not to be re-analysed in the results. Furthermore, the authors seem to apply the inclusion and exclusion criteria unsystematically when they declare that they only considered journal articles cited in HTA reports but then they present as a limitation that ‘For 2 reports (D06-01B and D06-01C), we also included evidence syntheses, although this document type was to be excluded.’ But fail to provide an explanation as to why such allowance was made.
In the results section two random examples are provided of HTA report, can the authors justify why these are explained in more detail and not the others? which criteria led the authors to provide these two examples? This feels selection bias and should be avoided.
In the Discussion section authors seem to contradict themselves. They start by saying that ‘the exclusion of studies published in non-English languages has only a minor effect on the conclusions of HTA reports’ At this point it is important to note that the measure of such effect (whether is big or small) is not provided throughout the manuscript nor it is explained how this ‘change in effect’ was estimated, there are no statistical tests and no p values provided making is really difficult to study whether the change, although minor, might or might not have been significant for the results of the HTA report and their future recommendations. The section follows by saying that ‘On the basis of our analysis, it is not possible to identify topics where non-English publications might be relevant.’ This statement seems to contradict what has just been said. And finally, they acknowledge that actually ‘it seems reasonable to conduct additional searches for non-English literature if a particular intervention is exclusively or preferentially available in a specific region’. The latter being a statement which defeats the point of the entire manuscript in its current rationale and supporting evidence.
Please fix reference number 1 to reflect the title (Cochrane Handbook for Systematic Reviews of Interventions) of the handbook in with Chapter 4 is included. Table 1 would benefit from including definitions for each of the categories. Figure 1 please include the distribution of the 38 reports as they are allocated to different categories. Table 5. What does EP mean?

Is the work clearly and accurately presented and does it cite the current literature?

No
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

No

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Information retrieval, systematic reviews, HTA, health research methods

CITE

Report a concern

Author Response 03 Apr 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

03 Apr 2025

Author Response

Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The first point raised is that ... Continue reading Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The first point raised is that the study is not suitable for making general statements about HTA reports.

Reply: This point was also raised by Reviewer 1 (see above). We have taken your concerns and suggestions and adapted the text

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to the German HTA reports

2. It was also criticized that the authors draw from one single study (Wu et al., 2009) to justify why non-English studies should no longer be searched for in HTA reports.

Reply: This is a misunderstanding, as Wu 2009 was cited primarily to show that the common assumption that foreign-language articles primarily produce negative/neutral results should be questioned. In order to strengthen the rationale for why we see our study as an important contribution, we have added a current systematic review in the background section, which we had originally only mentioned in the discussion.

Change in the text: In the background section, we have added the systematic review by Dobrescu et al. as a rationale for the present analysis. Please see changes in the text on page 4.

“However, a recent systematic review by Dobrescu et al. [9] found, that restricting evidence syntheses of interventions to English-language publications is a feasible methodological shortcut for most medical topics. The aim of our analysis was to examine whether the results of the systematic review are transferable to the context of HTA reports.”

3. Another comment refers to different information on the search / publication period in the abstract and the methods section.

Change to the text: We have now presented the publication period in the abstract.

“We determined whether non-English publications were included in all HTA reports on non-drug interventions (published by IQWiG between 06/2007 to 08/2018) and on selected drug interventions.”

4. It is also noted that it remains unclear whether HTA reports on drugs were included in the analysis.

Reply: Table 2 displays the number of included drug / non-drug HTA reports (3 HTA reports on drugs, 67 on non-drugs).

No change to the text

5. The reviewer recommends providing a detailed list of HTA reports (drug and non-drug) that were screened for inclusion in the methods section.

Reply: As stated in the methods, we included all HTA reports on non-drug interventions (published between 06/2007 and 08/2018) and selected 3 reports on drugs. We have added a list of analysed HTA reports to the supplement.

Change to the text: We have added a list of included HTA reports https://doi.org/10.5281/zenodo.12642959

6. The reviewer would like us to justify why diagnostic HTA reports were included from the beginning if they are not to be re-analysed in the results.

Reply: HTA reports on diagnostic questions can contain both results on the benefit of the diagnostic measure and on the diagnostic accuracy. Publications in other languages than English only contained information on diagnostic accuracy. However, in no case was a conclusion derived from this, but only the diagnostic accuracy was reported. Our article focuses on the impact of non-English publications on the conclusions of HTA reports, therefore a comparison of conclusions based on different data sets is not possible.

No change to the text

7. The reviewer is concerned that the inclusion and exclusion criteria were applied unsystematically and asks for an explanation as to why 2 HTA reports are cited under the limitations.

Reply: Based on your concerns, we have revised the extraction of the included studies in the HTA reports and have deleted the entries that did not fulfil the inclusion criteria (n = 11).

Change in the text: The tables and figures in the text have been adjusted accordingly. The total number of references included is now 2328, the number of non-English publications 126. Thus, we have deleted the corresponding section in the limitations.

8. In addition, the reviewer notes that there are two examples of HTA reports listed in the results section and asks for a more detailed explanation of why these were selected.

Reply: We wanted to illustrate a little more clearly to the reader why a non-English language article was categorized as A or B. As these are only examples, we see no risk of bias in the selection. The full list of HTA reports with non-English articles in categories A and B can be found in the supplement (https://doi.org/10.5281/zenodo.12642959).

No change to the text

9. The reviewer points out that the measure of effect is not explained and that no statistical tests and no p-values are given. In her view, this makes it difficult to analyse whether or not the change, even if small, could have been significant for the results of the HTA report and the future recommendations.

Reply: The measure of effect was defined as “change in the conclusion of the HTA report” which is composed of the conclusions of the individual endpoints. An overall conclusion was not provided by the underlying HTA reports. For most HTA reports (94%) no changes in the endpoints were identified. All changes were assessed in a yes/no format. This effect measure was therefore qualitative, even it was partly based on statistical analyses and re-analyses. The reviewer rightly points out that even small changes in a meta-analysis may or may not have been significant for the results of the HTA report. For this very reason, our analyses were not based on any statistical comparison between primary analysis and re-analysis. Rather, we examined whether the re-analysis led to a change in the conclusion of the endpoint and thus the HTA report, as this determines reimbursement and availability of health care services.

No change to the text

10. According to the reviewer, it is contradictory first to conclude that searching the non-English literature is generally unnecessary, but then to explain that such searches may be worthwhile for particular interventions that are available only a specific region.

Reply: We agree that a single HTA report on a country-specific intervention (N16-01) is a weak basis for recommending additional searches of non-English sources under specific circumstances. However, our text only describes this option as “reasonable” – without recommending it for all clinical interventions without clear indications that this is useful in the respective case.

Change in the text: We amended the text in the discussion

“This has less to do with a specific clinical question and more to do with market availability, which should of course be taken into account when preparing the systematic search.”

11. The reviewer asks to fix reference number 1 to reflect the title (Cochrane Handbook for Systematic Reviews of Interventions) of the handbook in which Chapter 4 is included.

Change in the text: We amended the citation

1. Lefebvre C, Glanville J, Briscoe S et al. Chapter 4: Searching for and selecting studies [last updated September 2024]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5 [online]. 2024 [Accessed: 20.01.2025]. URL: https://training.cochrane.org/handbook/current/chapter-04.

12. Table 1 would benefit from including definitions for each of the categories. You can find a description within the text (page 4): “Category A: change in the conclusion for a particular endpoint; Category B: no change in the conclusion for a particular endpoint; Category C: the non-English publications do not contain relevant information for the HTA report (e.g. because the amount or quality of the data is too low) and are therefore classified as “not relevant”.”

Change in the text: We have added the reference to Table 1 to the text

“The effect of the exclusion of non-English publications on conclusions was categorized for each HTA report as follows (see Table 1)”

13. The reviewer asks to include in Figure 1 the distribution of the 38 reports as they are allocated to different categories.

Reply: The requested information can be seen in Figure 1. As shown in the figure, 13 HTA reports contain only category C publications, 12 HTA reports with category B and 4 HTA reports to category A. The footnote also refers to the fact that an HTA report can contain non-English publications from different categories (A, B or C). The HTA report was assigned to the category that has the greatest impact on the change in conclusion. For example, the 12 HTA reports with category B also contain non-English publications from category C, but none from category A.

No change to the text

14. The reviewer asks what the abbreviation EP in Table 5 means.

Reply: This is the abbreviation for “endpoint“. An overview of the abbreviations used in the table is added below the table.

No change to the text
Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The first point raised is that the study is not suitable for making general statements about HTA reports.

Reply: This point was also raised by Reviewer 1 (see above). We have taken your concerns and suggestions and adapted the text

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to the German HTA reports

2. It was also criticized that the authors draw from one single study (Wu et al., 2009) to justify why non-English studies should no longer be searched for in HTA reports.

Reply: This is a misunderstanding, as Wu 2009 was cited primarily to show that the common assumption that foreign-language articles primarily produce negative/neutral results should be questioned. In order to strengthen the rationale for why we see our study as an important contribution, we have added a current systematic review in the background section, which we had originally only mentioned in the discussion.

Change in the text: In the background section, we have added the systematic review by Dobrescu et al. as a rationale for the present analysis. Please see changes in the text on page 4.

“However, a recent systematic review by Dobrescu et al. [9] found, that restricting evidence syntheses of interventions to English-language publications is a feasible methodological shortcut for most medical topics. The aim of our analysis was to examine whether the results of the systematic review are transferable to the context of HTA reports.”

3. Another comment refers to different information on the search / publication period in the abstract and the methods section.

Change to the text: We have now presented the publication period in the abstract.

“We determined whether non-English publications were included in all HTA reports on non-drug interventions (published by IQWiG between 06/2007 to 08/2018) and on selected drug interventions.”

4. It is also noted that it remains unclear whether HTA reports on drugs were included in the analysis.

Reply: Table 2 displays the number of included drug / non-drug HTA reports (3 HTA reports on drugs, 67 on non-drugs).

No change to the text

5. The reviewer recommends providing a detailed list of HTA reports (drug and non-drug) that were screened for inclusion in the methods section.

Reply: As stated in the methods, we included all HTA reports on non-drug interventions (published between 06/2007 and 08/2018) and selected 3 reports on drugs. We have added a list of analysed HTA reports to the supplement.

Change to the text: We have added a list of included HTA reports https://doi.org/10.5281/zenodo.12642959

6. The reviewer would like us to justify why diagnostic HTA reports were included from the beginning if they are not to be re-analysed in the results.

Reply: HTA reports on diagnostic questions can contain both results on the benefit of the diagnostic measure and on the diagnostic accuracy. Publications in other languages than English only contained information on diagnostic accuracy. However, in no case was a conclusion derived from this, but only the diagnostic accuracy was reported. Our article focuses on the impact of non-English publications on the conclusions of HTA reports, therefore a comparison of conclusions based on different data sets is not possible.

No change to the text

7. The reviewer is concerned that the inclusion and exclusion criteria were applied unsystematically and asks for an explanation as to why 2 HTA reports are cited under the limitations.

Reply: Based on your concerns, we have revised the extraction of the included studies in the HTA reports and have deleted the entries that did not fulfil the inclusion criteria (n = 11).

Change in the text: The tables and figures in the text have been adjusted accordingly. The total number of references included is now 2328, the number of non-English publications 126. Thus, we have deleted the corresponding section in the limitations.

8. In addition, the reviewer notes that there are two examples of HTA reports listed in the results section and asks for a more detailed explanation of why these were selected.

Reply: We wanted to illustrate a little more clearly to the reader why a non-English language article was categorized as A or B. As these are only examples, we see no risk of bias in the selection. The full list of HTA reports with non-English articles in categories A and B can be found in the supplement (https://doi.org/10.5281/zenodo.12642959).

No change to the text

9. The reviewer points out that the measure of effect is not explained and that no statistical tests and no p-values are given. In her view, this makes it difficult to analyse whether or not the change, even if small, could have been significant for the results of the HTA report and the future recommendations.

Reply: The measure of effect was defined as “change in the conclusion of the HTA report” which is composed of the conclusions of the individual endpoints. An overall conclusion was not provided by the underlying HTA reports. For most HTA reports (94%) no changes in the endpoints were identified. All changes were assessed in a yes/no format. This effect measure was therefore qualitative, even it was partly based on statistical analyses and re-analyses. The reviewer rightly points out that even small changes in a meta-analysis may or may not have been significant for the results of the HTA report. For this very reason, our analyses were not based on any statistical comparison between primary analysis and re-analysis. Rather, we examined whether the re-analysis led to a change in the conclusion of the endpoint and thus the HTA report, as this determines reimbursement and availability of health care services.

No change to the text

10. According to the reviewer, it is contradictory first to conclude that searching the non-English literature is generally unnecessary, but then to explain that such searches may be worthwhile for particular interventions that are available only a specific region.

Reply: We agree that a single HTA report on a country-specific intervention (N16-01) is a weak basis for recommending additional searches of non-English sources under specific circumstances. However, our text only describes this option as “reasonable” – without recommending it for all clinical interventions without clear indications that this is useful in the respective case.

Change in the text: We amended the text in the discussion

“This has less to do with a specific clinical question and more to do with market availability, which should of course be taken into account when preparing the systematic search.”

11. The reviewer asks to fix reference number 1 to reflect the title (Cochrane Handbook for Systematic Reviews of Interventions) of the handbook in which Chapter 4 is included.

Change in the text: We amended the citation

1. Lefebvre C, Glanville J, Briscoe S et al. Chapter 4: Searching for and selecting studies [last updated September 2024]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5 [online]. 2024 [Accessed: 20.01.2025]. URL: https://training.cochrane.org/handbook/current/chapter-04.

12. Table 1 would benefit from including definitions for each of the categories. You can find a description within the text (page 4): “Category A: change in the conclusion for a particular endpoint; Category B: no change in the conclusion for a particular endpoint; Category C: the non-English publications do not contain relevant information for the HTA report (e.g. because the amount or quality of the data is too low) and are therefore classified as “not relevant”.”

Change in the text: We have added the reference to Table 1 to the text

“The effect of the exclusion of non-English publications on conclusions was categorized for each HTA report as follows (see Table 1)”

13. The reviewer asks to include in Figure 1 the distribution of the 38 reports as they are allocated to different categories.

Reply: The requested information can be seen in Figure 1. As shown in the figure, 13 HTA reports contain only category C publications, 12 HTA reports with category B and 4 HTA reports to category A. The footnote also refers to the fact that an HTA report can contain non-English publications from different categories (A, B or C). The HTA report was assigned to the category that has the greatest impact on the change in conclusion. For example, the 12 HTA reports with category B also contain non-English publications from category C, but none from category A.

No change to the text

14. The reviewer asks what the abbreviation EP in Table 5 means.

Reply: This is the abbreviation for “endpoint“. An overview of the abbreviations used in the table is added below the table.

No change to the text
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 03 Apr 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

03 Apr 2025

Author Response

Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The first point raised is that ... Continue reading Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The first point raised is that the study is not suitable for making general statements about HTA reports.

Reply: This point was also raised by Reviewer 1 (see above). We have taken your concerns and suggestions and adapted the text

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to the German HTA reports

2. It was also criticized that the authors draw from one single study (Wu et al., 2009) to justify why non-English studies should no longer be searched for in HTA reports.

Reply: This is a misunderstanding, as Wu 2009 was cited primarily to show that the common assumption that foreign-language articles primarily produce negative/neutral results should be questioned. In order to strengthen the rationale for why we see our study as an important contribution, we have added a current systematic review in the background section, which we had originally only mentioned in the discussion.

Change in the text: In the background section, we have added the systematic review by Dobrescu et al. as a rationale for the present analysis. Please see changes in the text on page 4.

“However, a recent systematic review by Dobrescu et al. [9] found, that restricting evidence syntheses of interventions to English-language publications is a feasible methodological shortcut for most medical topics. The aim of our analysis was to examine whether the results of the systematic review are transferable to the context of HTA reports.”

3. Another comment refers to different information on the search / publication period in the abstract and the methods section.

Change to the text: We have now presented the publication period in the abstract.

“We determined whether non-English publications were included in all HTA reports on non-drug interventions (published by IQWiG between 06/2007 to 08/2018) and on selected drug interventions.”

4. It is also noted that it remains unclear whether HTA reports on drugs were included in the analysis.

Reply: Table 2 displays the number of included drug / non-drug HTA reports (3 HTA reports on drugs, 67 on non-drugs).

No change to the text

5. The reviewer recommends providing a detailed list of HTA reports (drug and non-drug) that were screened for inclusion in the methods section.

Reply: As stated in the methods, we included all HTA reports on non-drug interventions (published between 06/2007 and 08/2018) and selected 3 reports on drugs. We have added a list of analysed HTA reports to the supplement.

Change to the text: We have added a list of included HTA reports https://doi.org/10.5281/zenodo.12642959

6. The reviewer would like us to justify why diagnostic HTA reports were included from the beginning if they are not to be re-analysed in the results.

Reply: HTA reports on diagnostic questions can contain both results on the benefit of the diagnostic measure and on the diagnostic accuracy. Publications in other languages than English only contained information on diagnostic accuracy. However, in no case was a conclusion derived from this, but only the diagnostic accuracy was reported. Our article focuses on the impact of non-English publications on the conclusions of HTA reports, therefore a comparison of conclusions based on different data sets is not possible.

No change to the text

7. The reviewer is concerned that the inclusion and exclusion criteria were applied unsystematically and asks for an explanation as to why 2 HTA reports are cited under the limitations.

Reply: Based on your concerns, we have revised the extraction of the included studies in the HTA reports and have deleted the entries that did not fulfil the inclusion criteria (n = 11).

Change in the text: The tables and figures in the text have been adjusted accordingly. The total number of references included is now 2328, the number of non-English publications 126. Thus, we have deleted the corresponding section in the limitations.

8. In addition, the reviewer notes that there are two examples of HTA reports listed in the results section and asks for a more detailed explanation of why these were selected.

Reply: We wanted to illustrate a little more clearly to the reader why a non-English language article was categorized as A or B. As these are only examples, we see no risk of bias in the selection. The full list of HTA reports with non-English articles in categories A and B can be found in the supplement (https://doi.org/10.5281/zenodo.12642959).

No change to the text

9. The reviewer points out that the measure of effect is not explained and that no statistical tests and no p-values are given. In her view, this makes it difficult to analyse whether or not the change, even if small, could have been significant for the results of the HTA report and the future recommendations.

Reply: The measure of effect was defined as “change in the conclusion of the HTA report” which is composed of the conclusions of the individual endpoints. An overall conclusion was not provided by the underlying HTA reports. For most HTA reports (94%) no changes in the endpoints were identified. All changes were assessed in a yes/no format. This effect measure was therefore qualitative, even it was partly based on statistical analyses and re-analyses. The reviewer rightly points out that even small changes in a meta-analysis may or may not have been significant for the results of the HTA report. For this very reason, our analyses were not based on any statistical comparison between primary analysis and re-analysis. Rather, we examined whether the re-analysis led to a change in the conclusion of the endpoint and thus the HTA report, as this determines reimbursement and availability of health care services.

No change to the text

10. According to the reviewer, it is contradictory first to conclude that searching the non-English literature is generally unnecessary, but then to explain that such searches may be worthwhile for particular interventions that are available only a specific region.

Reply: We agree that a single HTA report on a country-specific intervention (N16-01) is a weak basis for recommending additional searches of non-English sources under specific circumstances. However, our text only describes this option as “reasonable” – without recommending it for all clinical interventions without clear indications that this is useful in the respective case.

Change in the text: We amended the text in the discussion

“This has less to do with a specific clinical question and more to do with market availability, which should of course be taken into account when preparing the systematic search.”

11. The reviewer asks to fix reference number 1 to reflect the title (Cochrane Handbook for Systematic Reviews of Interventions) of the handbook in which Chapter 4 is included.

Change in the text: We amended the citation

1. Lefebvre C, Glanville J, Briscoe S et al. Chapter 4: Searching for and selecting studies [last updated September 2024]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5 [online]. 2024 [Accessed: 20.01.2025]. URL: https://training.cochrane.org/handbook/current/chapter-04.

12. Table 1 would benefit from including definitions for each of the categories. You can find a description within the text (page 4): “Category A: change in the conclusion for a particular endpoint; Category B: no change in the conclusion for a particular endpoint; Category C: the non-English publications do not contain relevant information for the HTA report (e.g. because the amount or quality of the data is too low) and are therefore classified as “not relevant”.”

Change in the text: We have added the reference to Table 1 to the text

“The effect of the exclusion of non-English publications on conclusions was categorized for each HTA report as follows (see Table 1)”

13. The reviewer asks to include in Figure 1 the distribution of the 38 reports as they are allocated to different categories.

Reply: The requested information can be seen in Figure 1. As shown in the figure, 13 HTA reports contain only category C publications, 12 HTA reports with category B and 4 HTA reports to category A. The footnote also refers to the fact that an HTA report can contain non-English publications from different categories (A, B or C). The HTA report was assigned to the category that has the greatest impact on the change in conclusion. For example, the 12 HTA reports with category B also contain non-English publications from category C, but none from category A.

No change to the text

14. The reviewer asks what the abbreviation EP in Table 5 means.

Reply: This is the abbreviation for “endpoint“. An overview of the abbreviations used in the table is added below the table.

No change to the text
Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The first point raised is that the study is not suitable for making general statements about HTA reports.

Reply: This point was also raised by Reviewer 1 (see above). We have taken your concerns and suggestions and adapted the text

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to the German HTA reports

2. It was also criticized that the authors draw from one single study (Wu et al., 2009) to justify why non-English studies should no longer be searched for in HTA reports.

Reply: This is a misunderstanding, as Wu 2009 was cited primarily to show that the common assumption that foreign-language articles primarily produce negative/neutral results should be questioned. In order to strengthen the rationale for why we see our study as an important contribution, we have added a current systematic review in the background section, which we had originally only mentioned in the discussion.

Change in the text: In the background section, we have added the systematic review by Dobrescu et al. as a rationale for the present analysis. Please see changes in the text on page 4.

“However, a recent systematic review by Dobrescu et al. [9] found, that restricting evidence syntheses of interventions to English-language publications is a feasible methodological shortcut for most medical topics. The aim of our analysis was to examine whether the results of the systematic review are transferable to the context of HTA reports.”

3. Another comment refers to different information on the search / publication period in the abstract and the methods section.

Change to the text: We have now presented the publication period in the abstract.

“We determined whether non-English publications were included in all HTA reports on non-drug interventions (published by IQWiG between 06/2007 to 08/2018) and on selected drug interventions.”

4. It is also noted that it remains unclear whether HTA reports on drugs were included in the analysis.

Reply: Table 2 displays the number of included drug / non-drug HTA reports (3 HTA reports on drugs, 67 on non-drugs).

No change to the text

5. The reviewer recommends providing a detailed list of HTA reports (drug and non-drug) that were screened for inclusion in the methods section.

Reply: As stated in the methods, we included all HTA reports on non-drug interventions (published between 06/2007 and 08/2018) and selected 3 reports on drugs. We have added a list of analysed HTA reports to the supplement.

Change to the text: We have added a list of included HTA reports https://doi.org/10.5281/zenodo.12642959

6. The reviewer would like us to justify why diagnostic HTA reports were included from the beginning if they are not to be re-analysed in the results.

Reply: HTA reports on diagnostic questions can contain both results on the benefit of the diagnostic measure and on the diagnostic accuracy. Publications in other languages than English only contained information on diagnostic accuracy. However, in no case was a conclusion derived from this, but only the diagnostic accuracy was reported. Our article focuses on the impact of non-English publications on the conclusions of HTA reports, therefore a comparison of conclusions based on different data sets is not possible.

No change to the text

7. The reviewer is concerned that the inclusion and exclusion criteria were applied unsystematically and asks for an explanation as to why 2 HTA reports are cited under the limitations.

Reply: Based on your concerns, we have revised the extraction of the included studies in the HTA reports and have deleted the entries that did not fulfil the inclusion criteria (n = 11).

Change in the text: The tables and figures in the text have been adjusted accordingly. The total number of references included is now 2328, the number of non-English publications 126. Thus, we have deleted the corresponding section in the limitations.

8. In addition, the reviewer notes that there are two examples of HTA reports listed in the results section and asks for a more detailed explanation of why these were selected.

Reply: We wanted to illustrate a little more clearly to the reader why a non-English language article was categorized as A or B. As these are only examples, we see no risk of bias in the selection. The full list of HTA reports with non-English articles in categories A and B can be found in the supplement (https://doi.org/10.5281/zenodo.12642959).

No change to the text

9. The reviewer points out that the measure of effect is not explained and that no statistical tests and no p-values are given. In her view, this makes it difficult to analyse whether or not the change, even if small, could have been significant for the results of the HTA report and the future recommendations.

Reply: The measure of effect was defined as “change in the conclusion of the HTA report” which is composed of the conclusions of the individual endpoints. An overall conclusion was not provided by the underlying HTA reports. For most HTA reports (94%) no changes in the endpoints were identified. All changes were assessed in a yes/no format. This effect measure was therefore qualitative, even it was partly based on statistical analyses and re-analyses. The reviewer rightly points out that even small changes in a meta-analysis may or may not have been significant for the results of the HTA report. For this very reason, our analyses were not based on any statistical comparison between primary analysis and re-analysis. Rather, we examined whether the re-analysis led to a change in the conclusion of the endpoint and thus the HTA report, as this determines reimbursement and availability of health care services.

No change to the text

10. According to the reviewer, it is contradictory first to conclude that searching the non-English literature is generally unnecessary, but then to explain that such searches may be worthwhile for particular interventions that are available only a specific region.

Reply: We agree that a single HTA report on a country-specific intervention (N16-01) is a weak basis for recommending additional searches of non-English sources under specific circumstances. However, our text only describes this option as “reasonable” – without recommending it for all clinical interventions without clear indications that this is useful in the respective case.

Change in the text: We amended the text in the discussion

“This has less to do with a specific clinical question and more to do with market availability, which should of course be taken into account when preparing the systematic search.”

11. The reviewer asks to fix reference number 1 to reflect the title (Cochrane Handbook for Systematic Reviews of Interventions) of the handbook in which Chapter 4 is included.

Change in the text: We amended the citation

1. Lefebvre C, Glanville J, Briscoe S et al. Chapter 4: Searching for and selecting studies [last updated September 2024]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5 [online]. 2024 [Accessed: 20.01.2025]. URL: https://training.cochrane.org/handbook/current/chapter-04.

12. Table 1 would benefit from including definitions for each of the categories. You can find a description within the text (page 4): “Category A: change in the conclusion for a particular endpoint; Category B: no change in the conclusion for a particular endpoint; Category C: the non-English publications do not contain relevant information for the HTA report (e.g. because the amount or quality of the data is too low) and are therefore classified as “not relevant”.”

Change in the text: We have added the reference to Table 1 to the text

“The effect of the exclusion of non-English publications on conclusions was categorized for each HTA report as follows (see Table 1)”

13. The reviewer asks to include in Figure 1 the distribution of the 38 reports as they are allocated to different categories.

Reply: The requested information can be seen in Figure 1. As shown in the figure, 13 HTA reports contain only category C publications, 12 HTA reports with category B and 4 HTA reports to category A. The footnote also refers to the fact that an HTA report can contain non-English publications from different categories (A, B or C). The HTA report was assigned to the category that has the greatest impact on the change in conclusion. For example, the 12 HTA reports with category B also contain non-English publications from category C, but none from category A.

No change to the text

14. The reviewer asks what the abbreviation EP in Table 5 means.

Reply: This is the abbreviation for “endpoint“. An overview of the abbreviations used in the table is added below the table.

No change to the text
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 18 Nov 2024

Knut Sundell, Department of social work and criminology, University of Gävle, Gävle, Sweden

Not Approved

https://doi.org/10.5256/f1000research.166008.r332664

Different aspects of publication bias (positive outcome bias, time-lag bias, and place-of-publication bias) constitute serious threats to the validity of research synthesis. If research syntheses are based on a biased selection of the available research, there is an increased risk of producing misleading results. Awareness of these types of bias is important for reading systematic reviews and incorporating their findings into clinical practice and decision making.

The authors have screened 70 HTA reports from the German HTA agency, the Institute for Quality and Efficiency in Health Care (IQWiG) searching for reports that include non-English articles. If at least one non-English publication was included, the authors assessed whether the exclusion of non-English publications changed the conclusion. Of 70 HTA reports, 38 (54%) included non-English publications. The exclusion of non-English studies has only a minor effect on the conclusions of HTA reports

The authors conclude that, (1) in general, studies published in non-English languages have little influence on the conclusions of HTA reports, and (2) For the vast majority of topics, a language restriction to English seems justified.

These conclusions seem premature because of possible selection bias. We don’t know if the non-English articles in these reports constitute all relevant articles or a skewed sample (e.g., because of positive outcome biased, ). To be able to conclude that non-English articles have little influence on the report conclusions, the articles need to represent all relevant articles (i.e., no publication bias) which will require an extensive search for grey literature, including non-peer reviewed articles, the types of articles that the authors have excluded (i.e., conferences abstracts, evidence syntheses, clinical study reports, or registry entries).

The best we can say about the results is that non-English articles that are included in IQWiG’s reports may have had scant influence on the HTA-reports’ conclusions. This may be a thought-provoking result but hardly one to base policy on including or excluding non-English articles.

Furthermore, I suggest that the authors discuss the types of bias and their relative importance for the design of the study. The exclusion of non-English publications might be motivated, but the results might still be biased because of positive outcome bias, time-lag bias, place-of-publication bias or other types of bias?

There is no motivation for only including the German HTA organization. What are the pros and cons of this choice?

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

No
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

I cannot comment. A qualified statistician is required.
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

No

Competing Interests: No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

CITE

Report a concern

Author Response 03 Apr 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

03 Apr 2025

Author Response

Thank you very much for taking the time to review our paper. We address your comments below and believe that our replies resolve your concerns.

1. In your peer ... Continue reading Thank you very much for taking the time to review our paper. We address your comments below and believe that our replies resolve your concerns.

1. In your peer review you point out that “our conclusions seem premature because of possible selection bias“. It is unclear to us whether this refers to the selection of HTA reports or to the selection of primary studies into these HTA reports. Taking your argument in the first sense, we agree that our conclusions primarily refer to German HTA reports. It may well be that, due to different legal frameworks, HTA reports in other countries address other medical topics or address the same topics using other methods of information retrieval. To emphasize that these are the results of a German HTA agency, we have adapted the text accordingly.

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to German HTA reports

2. The issue of selection bias could also be understood in the sense of publication bias. The reviewer suspects that no difference between inclusion and exclusion of non-English sources was found, because relevant literature was not found by any of search approaches. According to the reviewer, this requires “an extensive search for grey literature, including non-peer reviewed articles, the types of articles that the authors have excluded (i.e., conferences abstracts, evidence syntheses, clinical study reports, or registry entries).“ As shown in Table 2, the information retrieval for IQWiG reports includes a number of databases (with a range of 3-23 per report). In addition, study registries were searched, reference lists screened, requests to authors were made and - for all reports but rapid reports - public hearings took place. Therefore, we assume the there is no issue with publication bias. Furthermore, the remaining risk of publication bias would be the nearly the same for both approaches, i.e. with or without non-English sources, because most conference abstracts, grey literature, and study reports will either be written in English or will be unusable for HTA purposes due to a lack of key study information.

No change to the text

3. Furthermore, you suggest that the authors discuss the types of bias and their relative importance for the design of the study. From our point of view, it is highly likely that all the forms of bias mentioned had no influence on the changes to the conclusions. Especially if non-significant results tend to end up in non-English journals, this would have resulted in apparent differences between those HTA reports that included non-English sources and those that did not.

No change to the text

Finally, you state that there is no motivation to include only reports prepared by the German HTA organisation. We disagree, because it requires detailed data to perform the reassessments using other language selection criteria. It would have required multinational cooperation and adequate funding to scale up the study to several HTA agencies and several languages. We also see no point in expanding the sample. Especially since, as mentioned in our discussion, major HTA agencies such as AHRQ, HAS and NICE also use some sort of language restriction in their reports.

No change to the text
Thank you very much for taking the time to review our paper. We address your comments below and believe that our replies resolve your concerns.

1. In your peer review you point out that “our conclusions seem premature because of possible selection bias“. It is unclear to us whether this refers to the selection of HTA reports or to the selection of primary studies into these HTA reports. Taking your argument in the first sense, we agree that our conclusions primarily refer to German HTA reports. It may well be that, due to different legal frameworks, HTA reports in other countries address other medical topics or address the same topics using other methods of information retrieval. To emphasize that these are the results of a German HTA agency, we have adapted the text accordingly.

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to German HTA reports

2. The issue of selection bias could also be understood in the sense of publication bias. The reviewer suspects that no difference between inclusion and exclusion of non-English sources was found, because relevant literature was not found by any of search approaches. According to the reviewer, this requires “an extensive search for grey literature, including non-peer reviewed articles, the types of articles that the authors have excluded (i.e., conferences abstracts, evidence syntheses, clinical study reports, or registry entries).“ As shown in Table 2, the information retrieval for IQWiG reports includes a number of databases (with a range of 3-23 per report). In addition, study registries were searched, reference lists screened, requests to authors were made and - for all reports but rapid reports - public hearings took place. Therefore, we assume the there is no issue with publication bias. Furthermore, the remaining risk of publication bias would be the nearly the same for both approaches, i.e. with or without non-English sources, because most conference abstracts, grey literature, and study reports will either be written in English or will be unusable for HTA purposes due to a lack of key study information.

No change to the text

3. Furthermore, you suggest that the authors discuss the types of bias and their relative importance for the design of the study. From our point of view, it is highly likely that all the forms of bias mentioned had no influence on the changes to the conclusions. Especially if non-significant results tend to end up in non-English journals, this would have resulted in apparent differences between those HTA reports that included non-English sources and those that did not.

No change to the text

Finally, you state that there is no motivation to include only reports prepared by the German HTA organisation. We disagree, because it requires detailed data to perform the reassessments using other language selection criteria. It would have required multinational cooperation and adequate funding to scale up the study to several HTA agencies and several languages. We also see no point in expanding the sample. Especially since, as mentioned in our discussion, major HTA agencies such as AHRQ, HAS and NICE also use some sort of language restriction in their reports.

No change to the text
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 03 Apr 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

03 Apr 2025

Author Response

Thank you very much for taking the time to review our paper. We address your comments below and believe that our replies resolve your concerns.

1. In your peer ... Continue reading Thank you very much for taking the time to review our paper. We address your comments below and believe that our replies resolve your concerns.

1. In your peer review you point out that “our conclusions seem premature because of possible selection bias“. It is unclear to us whether this refers to the selection of HTA reports or to the selection of primary studies into these HTA reports. Taking your argument in the first sense, we agree that our conclusions primarily refer to German HTA reports. It may well be that, due to different legal frameworks, HTA reports in other countries address other medical topics or address the same topics using other methods of information retrieval. To emphasize that these are the results of a German HTA agency, we have adapted the text accordingly.

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to German HTA reports

2. The issue of selection bias could also be understood in the sense of publication bias. The reviewer suspects that no difference between inclusion and exclusion of non-English sources was found, because relevant literature was not found by any of search approaches. According to the reviewer, this requires “an extensive search for grey literature, including non-peer reviewed articles, the types of articles that the authors have excluded (i.e., conferences abstracts, evidence syntheses, clinical study reports, or registry entries).“ As shown in Table 2, the information retrieval for IQWiG reports includes a number of databases (with a range of 3-23 per report). In addition, study registries were searched, reference lists screened, requests to authors were made and - for all reports but rapid reports - public hearings took place. Therefore, we assume the there is no issue with publication bias. Furthermore, the remaining risk of publication bias would be the nearly the same for both approaches, i.e. with or without non-English sources, because most conference abstracts, grey literature, and study reports will either be written in English or will be unusable for HTA purposes due to a lack of key study information.

No change to the text

3. Furthermore, you suggest that the authors discuss the types of bias and their relative importance for the design of the study. From our point of view, it is highly likely that all the forms of bias mentioned had no influence on the changes to the conclusions. Especially if non-significant results tend to end up in non-English journals, this would have resulted in apparent differences between those HTA reports that included non-English sources and those that did not.

No change to the text

Finally, you state that there is no motivation to include only reports prepared by the German HTA organisation. We disagree, because it requires detailed data to perform the reassessments using other language selection criteria. It would have required multinational cooperation and adequate funding to scale up the study to several HTA agencies and several languages. We also see no point in expanding the sample. Especially since, as mentioned in our discussion, major HTA agencies such as AHRQ, HAS and NICE also use some sort of language restriction in their reports.

No change to the text
Thank you very much for taking the time to review our paper. We address your comments below and believe that our replies resolve your concerns.

1. In your peer review you point out that “our conclusions seem premature because of possible selection bias“. It is unclear to us whether this refers to the selection of HTA reports or to the selection of primary studies into these HTA reports. Taking your argument in the first sense, we agree that our conclusions primarily refer to German HTA reports. It may well be that, due to different legal frameworks, HTA reports in other countries address other medical topics or address the same topics using other methods of information retrieval. To emphasize that these are the results of a German HTA agency, we have adapted the text accordingly.

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to German HTA reports

2. The issue of selection bias could also be understood in the sense of publication bias. The reviewer suspects that no difference between inclusion and exclusion of non-English sources was found, because relevant literature was not found by any of search approaches. According to the reviewer, this requires “an extensive search for grey literature, including non-peer reviewed articles, the types of articles that the authors have excluded (i.e., conferences abstracts, evidence syntheses, clinical study reports, or registry entries).“ As shown in Table 2, the information retrieval for IQWiG reports includes a number of databases (with a range of 3-23 per report). In addition, study registries were searched, reference lists screened, requests to authors were made and - for all reports but rapid reports - public hearings took place. Therefore, we assume the there is no issue with publication bias. Furthermore, the remaining risk of publication bias would be the nearly the same for both approaches, i.e. with or without non-English sources, because most conference abstracts, grey literature, and study reports will either be written in English or will be unusable for HTA purposes due to a lack of key study information.

No change to the text

3. Furthermore, you suggest that the authors discuss the types of bias and their relative importance for the design of the study. From our point of view, it is highly likely that all the forms of bias mentioned had no influence on the changes to the conclusions. Especially if non-significant results tend to end up in non-English journals, this would have resulted in apparent differences between those HTA reports that included non-English sources and those that did not.

No change to the text

Finally, you state that there is no motivation to include only reports prepared by the German HTA organisation. We disagree, because it requires detailed data to perform the reassessments using other language selection criteria. It would have required multinational cooperation and adequate funding to scale up the study to several HTA agencies and several languages. We also see no point in expanding the sample. Especially since, as mentioned in our discussion, major HTA agencies such as AHRQ, HAS and NICE also use some sort of language restriction in their reports.

No change to the text
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 04 Oct 2024

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3	4	5
Version 3 (revision) 07 Jul 25			read
Version 2 (revision) 03 Apr 25				read	read
Version 1 04 Oct 24	read	read	read

Knut Sundell, University of Gävle, Gävle, Sweden
Sonia Garcia Gonzalez-Moral, Newcastle University, Newcastle upon Tyne, UK
Tarquin Mittermayr, Austrian Institute for Health Technology Assessments, Wien, Austria
Steve McDonald, Monash University, Clayton, Australia
Liz Dennett, University of Alberta, Edmonton, Canada

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

1 Views

18 Aug 2025 | for Version 3

Tarquin Mittermayr, Austrian Institute for Health Technology Assessments, Wien, Austria

1 Views Cite this report Responses(0)

Approved

Thank you for providing me with another opportunity to review the revised versions of this paper.

I am happy with the way the authors have taken into consideration the reviewers' comments and the changes they have made accordingly. Furthermore, they have provided comprehensible arguments where they did not see a reason for changing the text.

I agree with the convincing comment of another reviewer that this paper provides sound arguments for the readers to make their own assessment of the impact that limiting searches to contain only English-language publications may have.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Conducting systematic searches in the field of Evidence Based Medicine and Health Technology Assessment

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

23 Views

24 Apr 2025 | for Version 2

Liz Dennett, University of Alberta, Edmonton, Alberta, Canada

23 Views Cite this report Responses(1)

Approved

This review reviewed 70 Health Technology Assessment reports published at the Institute for Quality and Efficiency in Health Care in Germany between 2006 and 2018. They chose to mainly include (96%) non-drug HTA's because they felt those would be more likely to have non-english reports than the drug HTAs. Fifty four percent of these HTAs had english language studies included, but in only 4 HTA reports did the english language studies change results for any of the studies' endpoints. In the 4 HTA reports, 10 studies changed the conclusions on 13 endpoints. The changes were not dramatic ones but in the case of 5 endpoints, no conclusion was possible without the non-english studies as they were the only studies containing information on that endpoint. From these results the authors conclude: "In general, studies only published in non-English languages have little influence on the conclusions of German HTA reports. For the vast majority of topics, a language restriction to English seems justified". They continue to recommend searching for non-english studies in special cases (e.g. a device is only marketed in one country).
Overall, I believe this to be a well conducted study with transparent methods and it presents similar findings to a number of other published articles (all of the ones I am aware of are included in the Dobrescu, et al., 2021 (Ref 1) paper cited in this study.

I don't really have any comments that need to be addressed by the authors, but I can't help putting only a "Partly" for the question "Are the conclusions drawn adequately supported by the results? At the end of the day, you can't prove a negative, and it is impossible to definitively say, "It is completely fine to ignore non-english studies." This study does, however, add to the growing evidence that it is unlikely to matter that much. It is then up to individual teams if they are comfortable with the albeit small risk of losing important information for the feasibility trade off. When we train people to do evidence synthesis, we always remind students several times that the goal of every search is to be as comprehensive as possible. It is hard to go against that initial training, even when you come to appreciate that true comprehensiveness is rarely (if ever) possible. I also think it is important to keep in mind the caveat (that these authors include) that China's growing influence in scientific discourse, which means that we can't consider this topic completely settled.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Dobrescu AI, Nussbaumer-Streit B, Klerings I, Wagner G, et al.: Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review.J Clin Epidemiol. 2021; 137: 209-217 PubMed Abstract | Publisher Full Text

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Expert searching

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Back to all reports

Reviewer Report

10 Views

21 Apr 2025 | for Version 2

Steve McDonald, School of Public Health and Preventive Medicine, Monash University, Clayton, Victoria, Australia

10 Views Cite this report Responses(1)

Approved

Thank you for the opportunity to review this revised version. This paper is a useful contribution to the evidence base of how language bias may affect the results of evidence synthesis. The authors provide a comprehensive case study of the impact of excluding non-English language publications on the conclusions of 70 HTA reports produced by IQWIG from 2007 to 2018.

Previous peer reviewers raised several valid concerns and suggested amendments, notably that the conclusions of the study should be limited to the context of German HTA reports. I feel the authors have adequately addressed the concerns raised. The revised paper is well-written and clearly presented, with sufficient information provided to allow readers to make their own assessment of the value of restricting to the inclusion of English-language publications.

The authors' conclusion that non-English publications "have little influence" or "minor effect" is supported by the data. Further, the authors provide a useful caveat to when searching for (and including) non-English publications should be considered.

One brief comment: the analysis focuses on journal publications (which is reasonable). It is not clear if (for these 70 HTA reports) non-journal publications in languages other than English were included, and thus whether they had any impact on the the conclusions. I guess the question is to what extent the 'unnecessary' searching of non-English publications extends to non-journal article formats (conference abstracts, register entries).

One minor comment: the flow diagram (second row) has 128 non-English publications, but in the Abstract and Results this is 126.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

References

1. Hausner E, Sturtz S, Molnar S, Schell L, et al.: Searching for non-English literature may be unnecessary for German HTA Reports. F1000Research. 2025; 13. Publisher Full Text

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Information specialist with expertise in evidence synthesis.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

11 Sep 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, Germany

Thank you for taking the time to review our article. Unfortunately, we are unable to answer your question as to whether our conclusions can also be applied to other forms of publication such as conference abstracts, as this was not part of our original research question.
Your reference to the numbers in the flowchart is correct. The following data must be adjusted in the flowchart: 2,328 instead of 2,339 publications, and 126 instead of 128 non-English publications. We will update the flowchart accordingly.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

16 Views

28 Dec 2024 | for Version 1

Tarquin Mittermayr, Austrian Institute for Health Technology Assessments, Wien, Austria

16 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

I cannot comment. A qualified statistician is required.
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Conducting systematic searches in my position as an Information Specialist

Respond to this report

Responses (1)

Author Response

03 Apr 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The reviewer indicates that it would be helpful to explain or reference the internal protocol mentioned in the methods section.

Reply: We have exchanged the term “protocol” with “project outline”. This document is a German-language 1-page text that provides an overview of the project, not a complete step-by-step protocol. It was not published and we therefore cannot reference it.

Change in the text: We have amended the text
“Following an internal project outline, we screened HTA reports (all reports on non-drug interventions published by IQWiG between 06/2007 and 08/2018 and randomly selected reports on drugs published between 09/2011 and 11/2016) for the inclusion of non-English publications.”

2. The reviewer states that it would have been helpful to either provide an explanation already in this section or refer to the appropriate page (e.g. “please find details on page 8”) in the limitations section as to why no publications after 2018 were considered for this assessment.

Change in the text: We have amended the text

”For the present article, it was therefore not meaningful to update the pool of HTA reports by including HTA reports published by IQWiG after October 2019 (see the methods section for details).“

3. The reviewer states that table 1 would benefit from clearer and more concise information: The same reasons for classifying low quality studies as category B in the classification table (“Amount and quality of the evidence…”) are listed in the explanation of the categories below the table under category C, thus giving room for ambiguous interpretations.

Reply: We disagree that there is ambiguity, because in category B the reason is “Amount and quality of the evidence of all studies”, whereas in category C the reason is “Quality of study too low to change conclusion”. The first reason refers to the evidence in its totality, the second reason addresses the quality of only the non-English study.

No change to the text

4. The reviewer points out that in Table 2 it is unclear whether the number of included publications per report (22.5) is an average, median or percentage.

Reply: Numbers and percentages are indicated by round brackets, while median and ranges are reported with square brackets. The heading indicates that 22.5 [1; 156] is the median, with the range given. We believe that it is reasonable to display median and range, rather than just reporting the range.

No change to the text

5. The reviewer refers to the different information from Table 1 and the section ‘Limitations’ regarding the examination of study quality. In addition, the reviewer states that the study would have benefited from performing a qualitative analysis for publications.

Reply: We are uncertain what type of “qualitative analysis” could be useful to be performed in addition to analysing the available evidence. As HTA work is evidence-based, the methods and the results of the primary studies are the sole data sources. Unfortunately, a qualitative analysis was not possible due to the limited time available. However, this would be very interesting, as there are already several publications [1-3] on this topic that question the quality of non-English studies.

Change in the text: We have added the assessment of the risk of bias for the non-English studies in the text.

“However, all 10 non-English studies that changed the conclusions of an HTA report (see Table 5) were classified as highly biased in the underlying HTA reports.”

6. The reviewer points out that the example of N16-03 listed under ‘Results’ could lead to discomfort among experts in evidence-based medicine since it seems to undermine the core argument of this article.

Reply: Admittedly, the recommendation not to search for non-English literature would have been easier if the decision to include or exclude such sources had not affected the conclusion of any HTA report. As 4 reports (6%) experienced a change in the conclusions, our conclusion is already somewhat toned down (“seems justified”). We agree with the reviewer that the decision is not straightforward. Some experts will continue to search non-English sources, because they consider the risk of overlooking a study more important than the resources required for searching for non-English publications. In our view, however, the main conclusion is correctly inferred from our results.

No change to the text

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

21 Views

14 Dec 2024 | for Version 1

Sonia Garcia Gonzalez-Moral, Newcastle University, Newcastle upon Tyne, England, UK

21 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

No
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

No

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Information retrieval, systematic reviews, HTA, health research methods

Respond to this report

Responses (1)

Author Response

03 Apr 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

Thank you for your very helpful comments. We address all your points below and have revised the relevant sections in the manuscript.

1. The first point raised is that the study is not suitable for making general statements about HTA reports.

Reply: This point was also raised by Reviewer 1 (see above). We have taken your concerns and suggestions and adapted the text

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to the German HTA reports

2. It was also criticized that the authors draw from one single study (Wu et al., 2009) to justify why non-English studies should no longer be searched for in HTA reports.

Reply: This is a misunderstanding, as Wu 2009 was cited primarily to show that the common assumption that foreign-language articles primarily produce negative/neutral results should be questioned. In order to strengthen the rationale for why we see our study as an important contribution, we have added a current systematic review in the background section, which we had originally only mentioned in the discussion.

Change in the text: In the background section, we have added the systematic review by Dobrescu et al. as a rationale for the present analysis. Please see changes in the text on page 4.

“However, a recent systematic review by Dobrescu et al. [9] found, that restricting evidence syntheses of interventions to English-language publications is a feasible methodological shortcut for most medical topics. The aim of our analysis was to examine whether the results of the systematic review are transferable to the context of HTA reports.”

3. Another comment refers to different information on the search / publication period in the abstract and the methods section.

Change to the text: We have now presented the publication period in the abstract.

“We determined whether non-English publications were included in all HTA reports on non-drug interventions (published by IQWiG between 06/2007 to 08/2018) and on selected drug interventions.”

4. It is also noted that it remains unclear whether HTA reports on drugs were included in the analysis.

Reply: Table 2 displays the number of included drug / non-drug HTA reports (3 HTA reports on drugs, 67 on non-drugs).

No change to the text

5. The reviewer recommends providing a detailed list of HTA reports (drug and non-drug) that were screened for inclusion in the methods section.

Reply: As stated in the methods, we included all HTA reports on non-drug interventions (published between 06/2007 and 08/2018) and selected 3 reports on drugs. We have added a list of analysed HTA reports to the supplement.

Change to the text: We have added a list of included HTA reports https://doi.org/10.5281/zenodo.12642959

6. The reviewer would like us to justify why diagnostic HTA reports were included from the beginning if they are not to be re-analysed in the results.

Reply: HTA reports on diagnostic questions can contain both results on the benefit of the diagnostic measure and on the diagnostic accuracy. Publications in other languages than English only contained information on diagnostic accuracy. However, in no case was a conclusion derived from this, but only the diagnostic accuracy was reported. Our article focuses on the impact of non-English publications on the conclusions of HTA reports, therefore a comparison of conclusions based on different data sets is not possible.

No change to the text

7. The reviewer is concerned that the inclusion and exclusion criteria were applied unsystematically and asks for an explanation as to why 2 HTA reports are cited under the limitations.

Reply: Based on your concerns, we have revised the extraction of the included studies in the HTA reports and have deleted the entries that did not fulfil the inclusion criteria (n = 11).

Change in the text: The tables and figures in the text have been adjusted accordingly. The total number of references included is now 2328, the number of non-English publications 126. Thus, we have deleted the corresponding section in the limitations.

8. In addition, the reviewer notes that there are two examples of HTA reports listed in the results section and asks for a more detailed explanation of why these were selected.

Reply: We wanted to illustrate a little more clearly to the reader why a non-English language article was categorized as A or B. As these are only examples, we see no risk of bias in the selection. The full list of HTA reports with non-English articles in categories A and B can be found in the supplement (https://doi.org/10.5281/zenodo.12642959).

No change to the text

9. The reviewer points out that the measure of effect is not explained and that no statistical tests and no p-values are given. In her view, this makes it difficult to analyse whether or not the change, even if small, could have been significant for the results of the HTA report and the future recommendations.

Reply: The measure of effect was defined as “change in the conclusion of the HTA report” which is composed of the conclusions of the individual endpoints. An overall conclusion was not provided by the underlying HTA reports. For most HTA reports (94%) no changes in the endpoints were identified. All changes were assessed in a yes/no format. This effect measure was therefore qualitative, even it was partly based on statistical analyses and re-analyses. The reviewer rightly points out that even small changes in a meta-analysis may or may not have been significant for the results of the HTA report. For this very reason, our analyses were not based on any statistical comparison between primary analysis and re-analysis. Rather, we examined whether the re-analysis led to a change in the conclusion of the endpoint and thus the HTA report, as this determines reimbursement and availability of health care services.

No change to the text

10. According to the reviewer, it is contradictory first to conclude that searching the non-English literature is generally unnecessary, but then to explain that such searches may be worthwhile for particular interventions that are available only a specific region.

Reply: We agree that a single HTA report on a country-specific intervention (N16-01) is a weak basis for recommending additional searches of non-English sources under specific circumstances. However, our text only describes this option as “reasonable” – without recommending it for all clinical interventions without clear indications that this is useful in the respective case.

Change in the text: We amended the text in the discussion

“This has less to do with a specific clinical question and more to do with market availability, which should of course be taken into account when preparing the systematic search.”

11. The reviewer asks to fix reference number 1 to reflect the title (Cochrane Handbook for Systematic Reviews of Interventions) of the handbook in which Chapter 4 is included.

Change in the text: We amended the citation

1. Lefebvre C, Glanville J, Briscoe S et al. Chapter 4: Searching for and selecting studies [last updated September 2024]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5 [online]. 2024 [Accessed: 20.01.2025]. URL: https://training.cochrane.org/handbook/current/chapter-04.

12. Table 1 would benefit from including definitions for each of the categories. You can find a description within the text (page 4): “Category A: change in the conclusion for a particular endpoint; Category B: no change in the conclusion for a particular endpoint; Category C: the non-English publications do not contain relevant information for the HTA report (e.g. because the amount or quality of the data is too low) and are therefore classified as “not relevant”.”

Change in the text: We have added the reference to Table 1 to the text

“The effect of the exclusion of non-English publications on conclusions was categorized for each HTA report as follows (see Table 1)”

13. The reviewer asks to include in Figure 1 the distribution of the 38 reports as they are allocated to different categories.

Reply: The requested information can be seen in Figure 1. As shown in the figure, 13 HTA reports contain only category C publications, 12 HTA reports with category B and 4 HTA reports to category A. The footnote also refers to the fact that an HTA report can contain non-English publications from different categories (A, B or C). The HTA report was assigned to the category that has the greatest impact on the change in conclusion. For example, the 12 HTA reports with category B also contain non-English publications from category C, but none from category A.

No change to the text

14. The reviewer asks what the abbreviation EP in Table 5 means.

Reply: This is the abbreviation for “endpoint“. An overview of the abbreviations used in the table is added below the table.

No change to the text

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

35 Views

18 Nov 2024 | for Version 1

Knut Sundell, Department of social work and criminology, University of Gävle, Gävle, Sweden

35 Views Cite this report Responses(1)

Not Approved

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

No
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

I cannot comment. A qualified statistician is required.
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

No

Competing Interests

No competing interests were disclosed.

Respond to this report

Responses (1)

Author Response

03 Apr 2025

Elke Hausner, Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, 50670, Germany

Thank you very much for taking the time to review our paper. We address your comments below and believe that our replies resolve your concerns.

1. In your peer review you point out that “our conclusions seem premature because of possible selection bias“. It is unclear to us whether this refers to the selection of HTA reports or to the selection of primary studies into these HTA reports. Taking your argument in the first sense, we agree that our conclusions primarily refer to German HTA reports. It may well be that, due to different legal frameworks, HTA reports in other countries address other medical topics or address the same topics using other methods of information retrieval. To emphasize that these are the results of a German HTA agency, we have adapted the text accordingly.

Change in the text: instead of referring to HTA reports in general, the text now refers specifically to German HTA reports

2. The issue of selection bias could also be understood in the sense of publication bias. The reviewer suspects that no difference between inclusion and exclusion of non-English sources was found, because relevant literature was not found by any of search approaches. According to the reviewer, this requires “an extensive search for grey literature, including non-peer reviewed articles, the types of articles that the authors have excluded (i.e., conferences abstracts, evidence syntheses, clinical study reports, or registry entries).“ As shown in Table 2, the information retrieval for IQWiG reports includes a number of databases (with a range of 3-23 per report). In addition, study registries were searched, reference lists screened, requests to authors were made and - for all reports but rapid reports - public hearings took place. Therefore, we assume the there is no issue with publication bias. Furthermore, the remaining risk of publication bias would be the nearly the same for both approaches, i.e. with or without non-English sources, because most conference abstracts, grey literature, and study reports will either be written in English or will be unusable for HTA purposes due to a lack of key study information.

No change to the text

3. Furthermore, you suggest that the authors discuss the types of bias and their relative importance for the design of the study. From our point of view, it is highly likely that all the forms of bias mentioned had no influence on the changes to the conclusions. Especially if non-significant results tend to end up in non-English journals, this would have resulted in apparent differences between those HTA reports that included non-English sources and those that did not.

No change to the text

Finally, you state that there is no motivation to include only reports prepared by the German HTA organisation. We disagree, because it requires detailed data to perform the reassessments using other language selection criteria. It would have required multinational cooperation and adequate funding to scale up the study to several HTA agencies and several languages. We also see no point in expanding the sample. Especially since, as mentioned in our discussion, major HTA agencies such as AHRQ, HAS and NICE also use some sort of language restriction in their reports.

No change to the text

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Lefebvre C, Glanville J, Briscoe S, et al.: Chapter 4: Searching for and selecting studies.2023. [Accessed: 07.02.2024]. Reference Source

[2] 2. EUnetHTA JA3WP6B2-2 Authoring Team: Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness. Methodological Guidelines.2019. [Accessed: 06.02.2024]. Reference Source

[3] 3. Relevo R, Balshem H: Finding evidence for comparing medical interventions: AHRQ and the Effective Health Care Program. J. Clin. Epidemiol. 2011; 64(11): 1168–1177. PubMed Abstract | Publisher Full Text

[4] 4. Egger M, Zellweger-Zahner T, Schneider M, et al.: Language bias in randomised controlled trials published in English and German. Lancet. 1997; 350(9074): 326–329. Publisher Full Text

[5] 5. Wu T, Li Y, Bian Z, et al.: Randomized trials published in some Chinese journals: how many are randomized? Trials. 2009; 10: 46. PubMed Abstract | Publisher Full Text | Free Full Text

[6] 6. Jackson JL, Kuriyama A: How Often Do Systematic Reviews Exclude Articles Not Published in English? J. Gen. Intern. Med. 2019; 34(8): 1388–1389. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Hoffmann F, Allers K, Rombey T, et al.: Nearly 80 systematic reviews were published each day: Observational study on trends in epidemiology and reporting over the years 2000-2019. J. Clin. Epidemiol. 2021; 138: 1–11. PubMed Abstract | Publisher Full Text

[8] 8. Morrison A, Moulton K, Clark M, et al.: English-Language Restriction When Conducting Systematic Review-based Meta-analyses: Systematic Review of Published Studies.2009. [Accessed: 06.02.2024]. Reference Source

[9] 9. Institute for Quality and Efficiency for in Health Care: IQWiG.2024. [Accessed: 08.01.2024]. Reference Source

[10] 10. Institute for Quality and Efficiency for in Health Care: General Methods. [Accessed: 08.01.2024]. Reference Source

[11] 11. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen: Tiotropiumbromid bei COPD; Abschlussbericht.2012. [Accessed: 06.02.2024]. Reference Source

[12] 12. Institute for Quality and Efficiency in Health Care: Tiotropium bromide for COPD; Executive Summary.2012. [Accessed: 06.02.2024]. Reference Source

[13] 13. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen: Systemische Therapie bei Erwachsenen als Psychotherapieverfahren; Abschlussbericht.2017. [Accessed: 06.02.2024]. Reference Source

[14] 14. Institute for Quality and Efficiency in Health Care: Systemic therapy in adults as a psychotherapeutic approach; Extract.2017. [Accessed: 06.02.2024]. Reference Source

[15] 15. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen: Aktive Kniebewegungsschienen in der Behandlung von Rupturen des vorderen Kreuzbands; Abschlussbericht.2017. [Accessed: 06.02.2024]. Reference Source

[16] 16. Institute for Quality and Efficiency in Health Care: Active knee motion devices in the treatment of anterior cruciate ligament ruptures; Extract.2017. [Accessed: 06.02.2024]. Reference Source

[17] 17. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen: Einsatz von motorbetriebenen Bewegungsschienen (CPM) nach Interventionen am Knie- oder am Schultergelenk; Abschlussbericht.2018. [Accessed: 06.02.2024]. Reference Source

[18] 18. Institute for Quality and Efficiency for in Health Care: Motor-driven continuous passive motion (CPM) devices after interventions in the knee and shoulder joint; Extract.2018. [Accessed: 06.02.2024]. Reference Source

[19] 19. Dobrescu AI, Nussbaumer-Streit B, Klerings I, et al.: Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review. J. Clin. Epidemiol. 2021; 137: 209–217. PubMed Abstract | Publisher Full Text

[20] 20. Xue J, Chen W, Chen L, et al.: Significant discrepancies were found in pooled estimates of searching with Chinese indexes versus searching with English indexes. J. Clin. Epidemiol. 2016; 70: 246–253. PubMed Abstract | Publisher Full Text

[21] 21. Cohen JF, Korevaar DA, Wang J, et al.: Should we search Chinese biomedical databases when performing systematic reviews? Syst. Rev. 2015; 4(1): 23. PubMed Abstract | Publisher Full Text | Free Full Text

[22] 22. Wilson M, Wang Z, Dobler CC, et al.: Noninvasive Positive Pressure Ventilation in the Home (with addendum).2020. [Accessed: 06.02.2024]. Reference Source

[23] 23. National Institute for Health and Care Excellence: Remsima (infliximab biosimilar) for subcutaneous injection for managing Crohn’s disease and ulcerative colitis; Evidence review.2021. [Accessed: 06.02.2024]. Reference Source

[24] 24. National Institute for Health and Care Excellence: Antimicrobial prescribing: delafloxacin for community-acquired pneumonia; Evidence review.2021. [Accessed: 06.02.2024]. Reference Source

[25] 25. Haute Autorité de santé: Suivi par télésurveillance des patients porteurs d’un moniteur cardiaque implantable.2021. [Accessed: 06.02.2024]. Reference Source

[26] 26. Haute Autorité de santé: Evaluation du parage de plaie de la pulpe par coiffage pulpaire direct.2019. [Accessed: 06.02.2024]. Reference Source

[27] 27. Farrell K, Horton J: Airway Pressure Release Ventilation for Acute Respiratory Distress Syndrome: An Update. Can. J. Health Technol. 2022; 2(9). Publisher Full Text

[28] 28. Chou R, Cantor A, Dana T, et al.: Statin Use for the Primary Prevention of Cardiovascular Disease in Adults: A Systematic Review for the U.S. Preventive Services Task Force.2022. [Accessed: 06.02.2024]. Reference Source

[29] 29. Barbara AM, Grobelna A: Real-Time Continuous Glucose Monitoring for People Living With Type 1 Diabetes. Can. J. Health Technol. 2022; 2(8). Publisher Full Text

[30] 30. Hausner E: Searching for non-English literature may be unnecessary for HTA Reports - supplemental material. Supplementary-material_matrix_V2. [Data set]. Zenodo. 2024. Reference Source

[31] 31. Hausner E: Searching for non-English literature may be unnecessary for HTA Reports - supplemental material. Supplementary-material_table. [Data set]. Zenodo. 2024. Publisher Full Text

Searching for non-English literature may be unnecessary for HTA Reports

Abstract

Background

Methods

Results

Conclusions

Keywords

Introduction

Aim

Methods

Table 1. Classification of non-English publication.

Analysis

Results

Figure 1. Flowchart on the presence and impact of non-English publications.

Table 2. Characteristics of the HTA reports.

Table 3. Number of non-English publications included per HTA report.

Table 4. Languages of non-English publications included in the HTA reports.

Table 5. HTA reports with endpoints that resulted in a change in the conclusions (Category A).

Figure 2. Report N16-03: Forest plot for the endpoint “pain”.

Figure 3. Report N09-01: Forest plot for the endpoint “symptom scores at 3 months”.

Discussion

Limitations

Conclusions

Ethics and consent

Data availability

Underlying data

Extended data

Acknowledgments

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated