Home Browse Using citation tracking for systematic literature searching -study...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Study Protocol

Using citation tracking for systematic literature searching - study protocol for a scoping review of methodological studies and an expert survey

[version 1; peer review: 2 approved with reservations]

Julian Hirt^1,2, Thomas Nordhausen², Christian Appenzeller-Herzog³, Hannah Ewald ³

PUBLISHED 01 Dec 2020

Author details Author details

¹ Institute of Applied Nursing Science, Department of Health, Eastern Switzerland University of Applied Sciences (formerly FHS St.Gallen), St.Gallen, Switzerland
² International Graduate Academy, Institute of Health and Nursing Science, Medical Faculty, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany
³ University Medical Library, University of Basel, Basel, Switzerland

Julian Hirt
Roles: Conceptualization, Investigation, Methodology, Project Administration, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Thomas Nordhausen
Roles: Conceptualization, Investigation, Methodology, Validation, Visualization, Writing – Review & Editing

Christian Appenzeller-Herzog
Roles: Conceptualization, Investigation, Methodology, Validation, Visualization, Writing – Review & Editing

Hannah Ewald
Roles: Conceptualization, Investigation, Methodology, Project Administration, Supervision, Validation, Visualization, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

This article is included in the Future of Research (FoR) collection.

Abstract

Background: Up-to-date guidance on comprehensive study identification for systematic reviews is crucial. According to current recommendations, systematic searching should combine electronic database searching with supplementary search methods. One such supplementary search method is citation tracking. It aims at collecting directly and/or indirectly cited and citing references from "seed references”. Tailored and evidence-guided recommendations concerning the use of citation tracking are strongly needed.

Objective: We intend to develop recommendations for the use of citation tracking in health-related systematic literature searching. Our study will be guided by the following research questions: What are the benefits of citation tracking for health-related systematic literature searching? Which perspectives and experiences do experts in the field of literature retrieval methods have with regard to citation tracking in health-related systematic literature searching?

Methods: Our study will have two parts: a scoping review and an expert survey. The scoping review aims at identifying methodological studies on benefits or problems of citation tracking in health-related systematic literature searching with no restrictions on study design, language, and publication date. We will perform database searching in MEDLINE, The Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science Core Collection, two information science databases, and free web searching. Two reviewers will independently assess full texts of selected abstracts. We will conduct direct backward and forward citation tracking on included articles. The results of the scoping review will inform our expert survey through which we aim to learn about experts΄ perspectives and experiences. We will narratively synthesize the results and derive recommendations for performing health-related systematic reviews.

Keywords

Citation Tracking, Literature Search, Supplementary Search, Methods, Scoping Review, Research Methodology, Survey, Systematic Review

Corresponding author: Hannah Ewald

Competing interests: No competing interests were disclosed.

Grant information: The author(s) declared that no grants were involved in supporting this work.

Copyright: © 2020 Hirt J et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Hirt J, Nordhausen T, Appenzeller-Herzog C and Ewald H. Using citation tracking for systematic literature searching - study protocol for a scoping review of methodological studies and an expert survey [version 1; peer review: 2 approved with reservations]. F1000Research 2020, 9:1386 (https://doi.org/10.12688/f1000research.27337.1) First published: 01 Dec 2020, 9:1386 (https://doi.org/10.12688/f1000research.27337.1) Latest published: 02 Sep 2021, 9:1386 (https://doi.org/10.12688/f1000research.27337.3)

Introduction

Systematic reviews are considered to be of high clinical and methodological importance as they help to derive recommendations for health care practice and future research^1–3. A comprehensive literature search aiming to identify the available evidence as completely as possible is the foundation of every systematic review^4–6. Due to an ever-growing research volume, lack of universal taxonomy and indexation, as well as extensive time requirements for identifying studies in a systematic way, effective search approaches are required^5,7,8. According to current recommendations, systematic search approaches should include both electronic database searching and one or several supplementary search methods⁹. Potential supplementary search methods include citation tracking, contacting study authors or experts, handsearching, trial register searching, and web searching¹⁰. In this study, we focus on citation tracking.

Citation tracking is an umbrella term for multiple methods which directly or indirectly collect related references from so called "seed references". These references are usually eligible for inclusion into the review and known at the start of the citation search^10–12. In the workflow of a systematic review, seed references for citation tracking often are records derived from screening bibliographic database search results. The taxonomy used to describe the principles of citation tracking is non-uniform and heterogeneous^13–16. Citation tracking methods are sub-categorized into direct and indirect citation tracking (Figure 1a). For direct citation tracking, the words "backward" and "forward" denote the directionality of tracking^13,17,18. Backward citation tracking is the oldest form of citation tracking. It aims at identifying references cited by a seed reference - which can easily be achieved by checking the reference list. Terms like "footnote chasing" or "reference list searching" are synonyms^6,13. In contrast, forward citation tracking or chaining aims at identifying citing references, i.e. references that cite a seed reference¹⁹. Indirect citation tracking describes the identification of (i) co-cited references or co-citations (i.e. other references cited by citing literature of a seed reference) and of (ii) co-citing references (i.e. references sharing references with a seed reference)^11,20. Direct and indirect citation relationships of references based on a seed reference are illustrated in Figure 1b. Both direct and indirect citation tracking may contain one or more layers of iteration. To this end, researchers may use newly retrieved references as new seed references.

Figure 1. Overview of citation tracking methods.

1a: Hierarchical illustration of different citation tracking methods; 1b: Direct and indirect citation relationships of references based on a seed reference. A → B denotes A cites B.

Direct backward citation tracking of cited references is currently the most common citation tracking method. However, recent guidance suggests that combining several citation tracking methods (e.g. screening cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing. It is quite possible that the added value of any form of citation tracking is not the same for all systematic reviews. It rather depends on a variety of factors. For instance, citation tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)^10,20,21. Hence, tailored and evidence-guided recommendations on the use of citation tracking are strongly needed. However, none of the current topical reviews has in fact systematically identified available evidence on the use and benefits of citation tracking in the context of systematic literature searching¹⁰.

Therefore, the aim of our study is to develop recommendations for the use of citation tracking in health-related systematic literature searching and to identify research gaps. Our study will be guided by the following two research questions:

- What are the benefits of citation tracking for health-related systematic literature searching?
- Which perspectives and experiences do experts have with regard to citation tracking in health-related systematic literature searching?

Protocol

This protocol is reported according to the “Preferred Reporting Items for Systematic review and Meta-Analysis Protocols” (PRISMA-P) checklist²². Details on the structure of our manuscript reporting related to PRISMA-P are provided elsewhere²³. Our study will have two parts: a scoping review and an expert survey. The scoping review has the objective to map the benefits or problems of citation tracking or, if that is not sufficiently informative, possible gaps in related research. The objective of the expert survey is to identify the perspectives, experiences and perceptions of experts in the field of literature retrieval methods with regard to citation tracking. Together, the scoping review and expert survey will serve as a basis to derive implications and recommendations for current practice and future research^24–26. For the scoping review, we will use the framework by Arksey and O’Malley^26,27 and the “Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews” (PRISMA-ScR)²⁸. Since an expert survey can be part of a scoping review, and the scoping review and the expert survey serve the same overall aim, we will follow methodological recommendations for scoping reviews^25,26 and the “Strengthening The Reporting of Observational Studies in Epidemiology” (STROBE) statement²⁹.

Scoping review

Eligibility criteria. We will include any study with a focus on citation tracking as a means of evidence retrieval method which exhibits one of the following criteria: benefits/problems and/or effectiveness of (i) citation tracking in general; (ii) different methods of citation tracking (e.g. backward vs. forward, direct vs. indirect); or (iii) technical uses of citation tracking (e.g. comparing platforms and/or tools, e.g. Scopus vs. Web of Science, Oyster, Voyster). Eligible studies need to have a health-related context. Studies without an explicitly specified research context are also eligible. There will be no restrictions on study design, language, and publication date.

We will exclude studies solely using citation tracking for evidence retrieval, e.g. a systematic review applying citation tracking as a supplementary search technique, or studies focussing on citation tracking as a means to explore network or citation impact (i.e. bibliometric analysis). Additionally, we will exclude methodological guidelines only mentioning or recommending citation tracking without describing or assessing it. Furthermore, we will exclude editorials, commentaries, letters and abstract-only publications. Table 1 illustrates our inclusion and exclusion criteria per domain.

Table 1. Inclusion and exclusion criteria.

Domain	Inclusion criteria	Exclusion criteria
Study focus	Any study with a focus on citation tracking as an evidence retrieval method AND one of the following criteria: - any study assessing benefits/problems and/or effectiveness of citation tracking - any study comparing different methods of citation tracking (e.g. backward vs. forward, direct vs. indirect) - any study assessing technical uses of citation tracking (e.g. comparing platforms and/or tools, e.g. Scopus vs. WoS, Oyster, Voyster, etc.)	Any study solely using citation tracking for evidence retrieval (e.g. a systematic review applying citation tracking as supplementary search technique) OR any study solely assessing benefits and/or use and/or effectiveness of citation tracking to explore a network or citation impact (i.e. bibliometric analysis) OR any study describing solely the method of citation tracking without further describing or assessing it (focus citation tracking, search methods, systematic reviews or other types of reviews), e.g. guidelines for developing search strategies or guidelines for systematic or other reviews
Research context	Health-related or not specified	Other research contexts
Language	All languages	-
Publication year	All publication years	-
Publication type	Any	Editorials Commentaries Letters Abstract-only publications

Information sources. We will search MEDLINE via Ovid, CINAHL (Cumulative Index to Nursing and Allied Health Literature), LLISFT (Library Literature & Information Science Full Text), LISTA (Library, Information Science & Technology Abstracts) via EBSCO, and the Web of Science Core Collection using database-specific search strategies. Additionally, we will perform free web searching via Google Scholar as well as direct forward and backward citation tracking of included studies. For citation tracking, we will use Scopus, as this database seems to cover the largest number of relevant citations for the purpose of our review³⁰. We will also contact librarians in the field of health sciences and information specialists through several mailing lists (e.g. Netzwerk Fachbibliotheken Gesundheit, Canadian Medical Libraries, Expertsearching, MEDBIB-L/German-speaking medical librarians, and EAHIL-list) to ask for further studies.

Search strategy. Due to a lack of adequate index terms, our search strategy will be based on textwords only. To determine frequently occurring terms for inclusion into the search strategy, we analysed keywords in the titles and abstracts of potentially relevant publications retrieved from preliminary searches and similar articles identified via PubMed by using a text mining approach³¹. We restricted parts of our textwords to the title field in order to avoid retrieving systematic reviews using citation tracking.

All authors contributed to the development of search strategies. HE and CAH are information specialists with a professional background in research; JH and TN are researchers experienced in the development of search strategies. HE drafted the search strategy. To uncover potential flaws, JH peer-checked the drafted search strategy and discussed it with HE.

Box 1 shows the final search strategy for MEDLINE in Ovid syntax. To use the strategy in other databases, we will translate it by means of Polyglot Search Translator³². CAH will conduct the searches and eliminate duplicates using the Bramer method³³. We will perform free web searching in Google Scholar using search terms from our database search.

Box 1. Search strategy for MEDLINE via Ovid

(reference list OR reference lists OR ((reference OR references OR citation OR citations OR co-citation OR co-citations) ADJ3 (search OR searches OR searching OR searched OR screen OR screening OR chain OR chains OR chaining OR check OR checking OR checked OR chased OR chasing OR tracking OR tracked OR harvesting OR tool OR tools OR backward OR forward)) OR ((cited OR citing OR cocited OR cociting OR co-cited OR co-citing) ADJ3 (references OR reference)) OR citation discovery tool OR cocitation OR co-citation OR cocitations OR co-citations OR co-cited OR backward chaining OR forward chaining OR snowball sampling OR snowballing OR footnote chasing OR berry picking OR cross references OR cross referencing OR cross-references OR cross-referencing OR citation activity OR citation activities OR citation analysis OR citation analyses OR citation network OR citation networks OR citation relationship OR citation relationships).ti OR (((((strategy OR strategies OR method* OR literature OR evidence OR additional OR complementary OR supplementary) ADJ3 (find OR finding OR search* OR retriev*)) OR (database ADJ2 combin*)).ti) AND ((search OR searches OR searching OR searched).ab))

Data management. A bibliography management tool (e.g. Citavi, EndNote) will be used to manage the number of reference retrievals throughout the study selection process. Furthermore, we will use specific tools for study selection that we describe below.

Selection of sources of evidence. After an initial calibration phase (TN, JH, HE), two authors (JH, TN) will independently screen titles, abstracts, and full texts using Rayyan³⁴. They will solve disagreements by third author arbitration (HE). To screen the results of the citation tracking step, we will consider ASReview, particularly if the number of references exceeds 1000. ASReview combines machine (deep) learning models on a set of eligible studies with active learning on manual selections during title-abstract screening to generate a relevancy-ranked abstract list and to save screening time. Should the tool prove to be beneficial for reducing the screening load, we will consider conducting a more sensitive database search at a later stage and screen additional results with ASReview.

Data charting process. We will pilot a standardised data extraction sheet approved by consensus among the authors. We will extract bibliographic and geographic data, design- and study-specific data as well as results. We aim for an iterative data extraction process, but in the final publication, we will provide a detailed overview of extracted data items. The first author (JH) will extract data and the second author (TN) will peer-check the extraction. We will solve disagreements by third author arbitration (HE).

Synthesis of results. One author (JH) will narratively summarise study characteristics and results based on the data extraction. Depending on the results, we will also chart them graphically.

Expert survey

Setting. The results of the scoping review will inform our online expert survey (see below). We intend to collect data from international experts in the field of literature retrieval methods. To design and develop the survey, we will use SosciSurvey³⁵. The survey language will be English.

Recruitment. Recruitment will be based on a triple approach. First, we will contact authors of pertinent articles identified during the literature search as well as experts from our professional networks. This "person-based" approach will help us to identify experts who authored papers, books, comments, and reviews in the field of citation tracking. We will ask the contacted persons to take part in the survey and to forward the invitation³⁶. Second, we will identify and contact relevant national and international networks as well as systematic review collaborations using institutional contacts (e.g. Cochrane, Joanna Briggs Institute, Campbell Collaboration, Institute of Medicine, expert information specialists, and Evidence Synthesis International). We will also use mailing lists (e.g. Netzwerk Fachbibliotheken Gesundheit, Canadian Medical Libraries, Expertsearching, MEDBIB-L/German-speaking medical librarians, and EAHIL-list). This "network-based approach" will allow us to reach experts in the field of literature retrieval methods who are potentially using citation tracking without necessarily being the authors of methodological studies (yet). Third, we will announce the expert survey on our project page on ResearchGate, aiming to recruit further experts missed by our person- and network-based approaches. By using this triple approach, we intend to recruit between 60 and 150 participants. The authors of this study will not participate in the survey.

Data collection. We will iteratively develop the survey questions during the scoping review process based on our findings. Broadly speaking, our survey will contain questions on the use of citation tracking, preferred taxonomy, experiences with citation tracking, as well as recommendations for future practice and research. We will collect participantsʼ sociodemographic data, e.g. professional education and background, current field of work, years of experience in literature searching and citation tracking. We will schedule approximately five weeks for participation. Table 2 illustrates our reminder strategy to reach as many experts as possible.

Table 2. Reminder strategy of the online survey.

Process and time	Person-based approach	Network-based approach	Announcement on ResearchGate
Survey setup	Invitation	Invitation	Invitation
Two weeks after	Reminder	-	-
Four weeks after	Reminder	Reminder	Update
Survey closing	-	-	Closing information

Note: Person-based approach: contacting national and international networks and systematic review collaborations using mailing lists.

Data analysis. To analyse the survey data, we will apply descriptive statistics based on frequencies, percentages, and cross tables. We will thematically categorise free text answers³⁷.

Ethical concerns. The online expert survey will contain introductory information on our aims, the survey itself, data management and security. Since we do not expect vulnerability on the part of participants, we will not apply for ethical approval of the expert survey. Taking part in the survey will indicate consent to participate.

Dissemination of results

Our dissemination strategy uses multiple ways to share our study results with academic stakeholders. The final review and expert survey will be published in an international open access journal relevant in the field of information retrieval. Additionally, we will discuss our results with experts at national and international conferences (e.g. conference of the German Network for Evidence-based Medicine (EbM-Netzwerk), conference of the European Association for Health Information and Libraries (EAHIL). To inform about our study results and publications, we will use Twitter, ResearchGate, and mailing lists from relevant stakeholders such as Netzwerk Fachbibliotheken Gesundheit, Canadian Medical Libraries, Expertsearching, MEDBIB-L/German-speaking medical librarians, and EAHIL-list.

Study status

We intend to start our search in November 2020 and expect to complete the study in April 2022.

Current study status: preliminary searches: yes; piloting of the study selection process: yes; formal screening of search results against eligibility criteria: no; data extraction: no; data analysis: no.

Conclusions

Depending on the available study landscape, missing pertinent evidence might have an impact on the validity of systematic reviews and, consequently, on the quality of clinical care^38–41. Therefore, authors of systematic reviews should conduct high quality literature searches aiming to detect all relevant evidence. Citation tracking may be a good way to complement electronic database searches and to broaden the scope of possible findings. Therefore, our study intends to provide literature- and expert-based recommendations on the use of citation tracking for systematic literature searching. Specifically, we will provide guidance on; (i) situations in which citation tracking is likely to be particularly effective, (ii) situations in which a particular method of citation tracking is specifically indicated, and on (iii) technical/methodological aspects and pitfalls. We will also point out (iv) areas requiring more research in order to provide recommendations. It is possible that the recommendations developed during this project in a health-related context may prove relevant also for other academic fields such as social or environmental sciences^9,42. Finally, tailored and evidence-based recommendations concerning the use of citation tracking for systematic literature searching may guide future steps in semi-automated and automated literature retrieval methods^43,44.

Data availability

Underlying data

No underlying data are associated with this article.

Reporting guidelines

Open Science Framework (OSF): PRISMA-P checklist for ‘Using citation tracking for systematic literature searching: study protocol for a scoping review of methodological studies and an expert survey’, https://doi.org/10.17605/OSF.IO/7ETYD²³.

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Faculty Opinions recommended

References

1. Ioannidis JPA: The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016; 94(3): 485–514. PubMed Abstract | Publisher Full Text | Free Full Text
2. Abbas Z, Raza S, Ejaz K: Systematic reviews and their role in evidence-informed health care. J Pak Med Assoc. 2008; 58(10): 561–67. PubMed Abstract
3. Gough D, Davies P, Jamtvedt G, et al.: Evidence Synthesis International (ESI): Position Statement. Syst Rev. 2020; 9(1): 155. PubMed Abstract | Publisher Full Text | Free Full Text
4. Sutton A, Clowes M, Preston L, et al.: Meeting the review family: exploring review types and associated information retrieval requirements. Health Info Libr J. 2019; 36(3): 202–22. PubMed Abstract | Publisher Full Text
5. Gusenbauer M, Haddaway NR: Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res Synth Methods. 2020; 11(2): 181–217. PubMed Abstract | Publisher Full Text | Free Full Text
6. Lefebvre C, Glanville J, Briscoe S, et al.: Searching for and selecting studies. In: Higgins JPT, Thomas J, eds. Cochrane Handbook for Systematic Reviews of Interventions: Version 6, 2nd edn. Hoboken: Wiley Online Library. 2019; 67–108. Publisher Full Text
7. McGowan J, Sampson M: Systematic reviews need systematic searchers. J Med Libr Assoc. 2005; 93(1): 74–80. PubMed Abstract | Free Full Text
8. de Souza Leão L, Eyal G: The rise of randomized controlled trials (RCTs) in international development in historical perspective. Theor Soc. 2019; 48(3): 383–418. Publisher Full Text
9. Cooper C, Booth A, Varley-Campbell J, et al.: Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol. 2018; 18(1): 85. PubMed Abstract | Publisher Full Text | Free Full Text
10. Cooper C, Booth A, Britten N, et al.: A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017; 6(1): 234. PubMed Abstract | Publisher Full Text | Free Full Text
11. Belter CW: Citation analysis as a literature search method for systematic reviews. Journal of the Association for Information Science and Technology. 2016; 67(11): 2766–77. Publisher Full Text
12. Hu X, Rousseau R, Chen J: On the definition of forward and backward citation generations. Journal of Informetrics. 2011; 5(1): 27–36. Publisher Full Text
13. Booth A: Unpacking your literature search toolbox: on search styles and tactics. Health Info Libr J. 2008; 25(4): 313–17. PubMed Abstract | Publisher Full Text
14. Saimbert MK, Fowler SA, Pierce J, et al.: Search Resources and Techniques to Maximize Search Efforts. In: Holly C, Salmond SW, Saimbert MK, eds. Comprehensive Systematic Review for Advanced Nursing Practice. New York: Springer Publishing Company 2016; 139–72.
15. Choong MK, Tsafnat G: Role of citation tracking in updating of systematic reviews. AMIA Jt Summits Transl Sci Proc. 2014; 2014: 18. PubMed Abstract | Free Full Text
16. Lowe J, Peters J, Shields B, et al.: Methods to update systematic literature searches: full update searching vs. forward citation chasing: A case study from a systematic review of diagnostic test accuracy. Exeter o J. Reference Source
17. Booth A: Innovative approaches to systematic reviewing. In: Levay P, Craven J, eds. Systematic Searching: Practical ideas for improving results. London: Facet Publishing. 2019; 25–50.
18. Lefebvre C, Glanville J, Briscoe S, et al.: Chapter 4: Searching for and selecting studies. Draft version (13 September 2018) 2018. Reference Source
19. Cribbin T: Augmenting Citation Chain Aggregation with Article Maps. CEUR Workshop Proceedings. 2014; 1311, (accessed 18 Aug 2020). Reference Source
20. Janssens ACJW, Gwinn M: Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol. 2015; 15: 84. PubMed Abstract | Publisher Full Text | Free Full Text
21. Linder SK, Kamath GR, Pratt GF, et al.: Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015; 68(4): 412–17. PubMed Abstract | Publisher Full Text | Free Full Text
22. Shamseer L, Moher D, Clarke M, et al.: Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015; 350: g7647. PubMed Abstract | Publisher Full Text
23. Hirt J: Using citation tracking for systematic literature searching (supplementary material). 2020,. http://www.doi.org/10.17605/OSF.IO/7ETYD
24. Ruiz-Perez I, Petrova D: Scoping reviews. Another way of literature review. Med Clin (Barc). 2019; 153(4): 165–68. PubMed Abstract | Publisher Full Text
25. Levac D, Colquhoun H, O'Brien KK: Scoping studies: advancing the methodology. Implement Sci. 2010; 5: 69. PubMed Abstract | Publisher Full Text | Free Full Text
26. Arksey H, O'Malley L: Scoping studies: towards a methodological framework. Int J Soc. 2005; 8(1): 19–32. Publisher Full Text
27. Munn Z, Peters MDJ, Stern C, et al.: Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018; 18(1): 143. PubMed Abstract | Publisher Full Text | Free Full Text
28. Tricco AC, Lillie E, Zarin W, et al.: PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018; 169(7): 467–73. PubMed Abstract | Publisher Full Text
29. von Elm E, Altman DG, Egger M, et al.: Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007; 335(7624): 806–08. PubMed Abstract | Publisher Full Text | Free Full Text
30. Li J, Burnham JF, Lemley T, et al.: Citation analysis: Comparison of Web of Science®, Scopus™, SciFinder®, and Google Scholar. J Med Libr Assoc. 2010; 7(3): 196–217. Publisher Full Text
31. Glanville J: Text mining for information specialists. Levay P Craven J eds. Systematic Searching: Practical ideas for improving results. London: Facet Publishing, 2019: 147–70. Reference Source
32. Clark JM, Sanders S, Carter M, et al.: Improving the translation of search strategies using the Polyglot Search Translator: a randomized controlled trial. JMLA. 2020; 108(2): 195–207. PubMed Abstract | Publisher Full Text | Free Full Text
33. Bramer WM, Giustini D, de Jonge GB, et al.: De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc. 2016; 104(3): 240–43. PubMed Abstract | Publisher Full Text | Free Full Text
34. Ouzzani M, Hammady H, Fedorowicz Z, et al.: Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016; 5: 210. Reference Source
35. SoSci Survey GmbH: SoSci Survey 2020. Accessed August 14, 2020. Reference Source
36. Cooper C, Garside R, Varley-Campbell J, et al.: "it has no meaning to me". How do researchers understand the effectiveness of literature searches? A qualitative analysis and preliminary typology of understandings. Res Syn Meth. 2020; 11(5): 627–640. Publisher Full Text
37. Saldaña J: The Coding Manual for Qualitative Researchers, 2nd edn. London: SAGE Publications, 2013. Reference Source
38. Sampson M, McGowan J: Errors in search strategies were identified by type and frequency. J Clin Epidemiol. 2006; 59(10): 1057–1063. PubMed Abstract | Publisher Full Text
39. Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R: Errors in search strategies used in systematic reviews and their effects on information retrieval. J Med Libr Assoc. 2019; 107(2): 210–21. PubMed Abstract | Publisher Full Text | Free Full Text
40. Nussbaumer-Streit B, Klerings I, Wagner G, et al.: Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study. J Clin Epidemiol. 2018; 102: 1–11. PubMed Abstract | Publisher Full Text
41. Ewald H, Klerings I, Wagner G, et al.: Abbreviated and comprehensive literature searches led to identical or very similar effect estimates: meta-epidemiological study. J Clin Epidemiol. 2020; 128: 1–12. PubMed Abstract | Publisher Full Text
42. James KL, Randall NP, Haddaway NR: A methodology for systematic mapping in environmental sciences. Environ Evid. 2016; 5: 7. Publisher Full Text
43. O'Connor AM, Glasziou P, Taylor M, et al.: A focus on cross-purpose tools, automated recognition of study design in multiple disciplines, and evaluation of automation tools: a summary of significant discussions at the fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR). Syst Rev. 2020; 9(1): 100. PubMed Abstract | Publisher Full Text | Free Full Text
44. Kohl C, McIntosh EJ, Unger S, et al.: Online tools supporting the conduct and reporting of systematic reviews and systematic maps: A case study on CADIMA and review of existing tools. Environ Evid. 2018; 7(1): 2420. Publisher Full Text

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 01 Dec 2020

Author details Author details

Julian Hirt
Roles: Conceptualization, Investigation, Methodology, Project Administration, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Thomas Nordhausen
Roles: Conceptualization, Investigation, Methodology, Validation, Visualization, Writing – Review & Editing

Christian Appenzeller-Herzog
Roles: Conceptualization, Investigation, Methodology, Validation, Visualization, Writing – Review & Editing

Hannah Ewald
Roles: Conceptualization, Investigation, Methodology, Project Administration, Supervision, Validation, Visualization, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Article Versions (3)

version 3

Revised

Published: 02 Sep 2021, 9:1386

https://doi.org/10.12688/f1000research.27337.3

version 2

Revised

Published: 04 Aug 2021, 9:1386

https://doi.org/10.12688/f1000research.27337.2

version 1

Published: 01 Dec 2020, 9:1386

https://doi.org/10.12688/f1000research.27337.1

© 2020 Hirt J et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Hirt J, Nordhausen T, Appenzeller-Herzog C and Ewald H. Using citation tracking for systematic literature searching - study protocol for a scoping review of methodological studies and an expert survey [version 1; peer review: 2 approved with reservations]. F1000Research 2020, 9:1386 (https://doi.org/10.12688/f1000research.27337.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 01 Dec 2020

Views

Reviewer Report 07 Apr 2021

David Moher, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada

Approved with Reservations

https://doi.org/10.5256/f1000research.30208.r81307

The authors are proposing a new guidance on citation tracking. They will achieve this by completing a scoping review and subsequently a survey. The authors propose using appropriate methods (and reporting) for conducting the scoping review. I found the methods ... Continue reading

A more conceptual question is to what degree this initiative overlaps and differs from PRISMA-S (Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. doi: 10.1186/s13643-020-01542-z)¹.
For example, item 5 of PRIMSA-S – Citation searching - states “Citation searching 5 Indicate whether cited references or citing references were examined, and describe any methods used for locating cited/citing references (e.g., browsing reference lists, using a citation index, setting up email alerts for references citing included studies).” I think this is important so as not to potentially confuse readers. Indeed, should the current initiative be seen as an implementation of PRISMA-S?
I had more difficulty understanding the proposed survey. For me, much of the methods were missing. I understand the recruitment (well reported). I would recommend the authors include the PRISMA-S group unless there is a lot of overlap between PRISMA-S and the other groups mentioned.
It was not clear to me how long the authors will make completing the survey– 15 minutes of 45 minutes? Similarly, will the respondents receive an incentive for completing the survey?
Why ‘simply’ a survey rather than a Delphi (or modified) approach? Will the survey (or Delphi) be pilot tested for question and response option clarity and language?
The data analysis describes cross tables. I assume the authors mean cross tabulations which can be more than descriptive. Will the authors be conducting Chi-square analysis or other analytical approaches of the cross tabulations (e.g., p values)?
The ethical concerns section is likely jurisdiction specific. In my setting ethics would be required. Can the authors explicitly indicate whether ethics is required or not. The section is currently vague on this critical issue.
For the survey, do the authors have an estimated sample size they are aiming for. Similarly, do the authors have an estimated response rate they are aiming for?

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Partly

References

1. Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, et al.: PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews.Syst Rev. 2021; 10 (1): 39 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Systematic reviews; open science; reporting guidelines.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 04 Aug 2021

Julian Hirt, Institute of Applied Nursing Science, Department of Health, Eastern Switzerland University of Applied Sciences (formerly FHS St.Gallen), St.Gallen, Switzerland

04 Aug 2021

Author Response

Dear David Moher

We are grateful for your reviewer report and, in particular, for your helpful suggestions to derive evidence-based recommendations from our study. Below, we provide a detailed ... Continue reading Dear David Moher

We are grateful for your reviewer report and, in particular, for your helpful suggestions to derive evidence-based recommendations from our study. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
You will see that in response to your comments, we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope that these changes fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: A more conceptual question is to what degree this initiative overlaps and differs from PRISMA-S (Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. doi: 10.1186/s13643-020-01542-z).

Authors’ response: PRISMA-S is a reporting guideline for systematic literature searches. However, it is lacking on precise recommendations how to report citation tracking methods and results. By conducting a Delphi study with selected experts in the field (see our answer below for details), we aim at developing more detailed recommendations on adequately reporting citation tracking methods as part of systematic literature searching. This may complement recommendations on reporting given by the PRISMA-S group or could even be integrated in future versions. For the Delphi, we will invite the PRISMA-S group to best use synergies and experiences.

Reviewer comment: For example, item 5 of PRIMSA-S – Citation searching - states “Citation searching 5 Indicate whether cited references or citing references were examined, and describe any methods used for locating cited/citing references (e.g., browsing reference lists, using a citation index, setting up email alerts for references citing included studies).” I think this is important so as not to potentially confuse readers. Indeed, should the current initiative be seen as an implementation of PRISMA-S?

Authors’ response: See answer above. An important detail for the reporting of our study: since we will use citation tracking as part of our information retrieval, we will use PRISMA-S to guide our reporting of the scoping review. In the revised version, we reference it accordingly: “We will document our search strategy according to PRISMA-S”.

Reviewer comment: I had more difficulty understanding the proposed survey. For me, much of the methods were missing. I understand the recruitment (well reported). I would recommend the authors include the PRISMA-S group unless there is a lot of overlap between PRISMA-S and the other groups mentioned.

Authors’ response: As detailed further below, the survey will be replaced by a Delphi study. So, we replaced the survey methods with more detailed methods for the Delphi study. Concerning PRISMA-S, we agree with the reviewer and will ask the PRISMA-S working group to participate in our Delphi study (see below).

Reviewer comment: It was not clear to me how long the authors will make completing the survey– 15 minutes of 45 minutes? Similarly, will the respondents receive an incentive for completing the survey?

Authors’ response: Following your recommendation below, we now plan to conduct an expert Delphi study containing multiple Delphi rounds. We expect that experts will invest around 30 to 90 minutes per Delphi round depending on the underpinning aim of the Delphi round as well as experts’ familiarity and experiences with the topic. Participants will not receive an incentive for participation.

Reviewer comment: Why ‘simply’ a survey rather than a Delphi (or modified) approach? Will the survey (or Delphi) be pilot tested for question and response option clarity and language?

Authors’ response: This is an important point and we thank the reviewer for raising it. We agree that a Delphi study containing several Delphi rounds is suitable to collect the perspectives of international experts on citation tracking, promote discussions on the topic as well as to derive consensus recommendations for future practice and research on the use of citation tracking in systematic literature searching for health-related topics. We subjected our protocol to a major revision based on this point and now outline the methods for the Delphi. As stated in the revised text, we will pilot test and discuss our Delphi items with a person experienced in literature searching but who is not an author and not involved in the Delphi study.

Reviewer comment: The data analysis describes cross tables. I assume the authors mean cross tabulations which can be more than descriptive. Will the authors be conducting Chi-square analysis or other analytical approaches of the cross tabulations (e.g., p values)?

Authors’ response: We revised the corresponding section to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: The ethical concerns section is likely jurisdiction specific. In my setting ethics would be required. Can the authors explicitly indicate whether ethics is required or not. The section is currently vague on this critical issue.

Authors’ response: We added details to the ethical concerns section. With regard to the Swiss Human Research Act, our research does not concern human diseases and the structure and function of the human body. We will therefore not apply for ethical approval of the expert Delphi study.

Reviewer comment: For the survey, do the authors have an estimated sample size they are aiming for. Similarly, do the authors have an estimated response rate they are aiming for?

Authors’ response: This answer concerns the recruitment of experts for the Delphi study, not the survey. We intend to recruit at least 15 participants by using our stepwise approach for recruitment, the person-based approach (contacting authors of pertinent articles identified during the literature search as well as experts from authors’ professional networks) and the organisation-based approach (contacting national and international organisations and systematic review collaborations).
Dear David Moher

We are grateful for your reviewer report and, in particular, for your helpful suggestions to derive evidence-based recommendations from our study. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
You will see that in response to your comments, we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope that these changes fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: A more conceptual question is to what degree this initiative overlaps and differs from PRISMA-S (Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. doi: 10.1186/s13643-020-01542-z).

Authors’ response: PRISMA-S is a reporting guideline for systematic literature searches. However, it is lacking on precise recommendations how to report citation tracking methods and results. By conducting a Delphi study with selected experts in the field (see our answer below for details), we aim at developing more detailed recommendations on adequately reporting citation tracking methods as part of systematic literature searching. This may complement recommendations on reporting given by the PRISMA-S group or could even be integrated in future versions. For the Delphi, we will invite the PRISMA-S group to best use synergies and experiences.

Reviewer comment: For example, item 5 of PRIMSA-S – Citation searching - states “Citation searching 5 Indicate whether cited references or citing references were examined, and describe any methods used for locating cited/citing references (e.g., browsing reference lists, using a citation index, setting up email alerts for references citing included studies).” I think this is important so as not to potentially confuse readers. Indeed, should the current initiative be seen as an implementation of PRISMA-S?

Authors’ response: See answer above. An important detail for the reporting of our study: since we will use citation tracking as part of our information retrieval, we will use PRISMA-S to guide our reporting of the scoping review. In the revised version, we reference it accordingly: “We will document our search strategy according to PRISMA-S”.

Reviewer comment: I had more difficulty understanding the proposed survey. For me, much of the methods were missing. I understand the recruitment (well reported). I would recommend the authors include the PRISMA-S group unless there is a lot of overlap between PRISMA-S and the other groups mentioned.

Authors’ response: As detailed further below, the survey will be replaced by a Delphi study. So, we replaced the survey methods with more detailed methods for the Delphi study. Concerning PRISMA-S, we agree with the reviewer and will ask the PRISMA-S working group to participate in our Delphi study (see below).

Reviewer comment: It was not clear to me how long the authors will make completing the survey– 15 minutes of 45 minutes? Similarly, will the respondents receive an incentive for completing the survey?

Authors’ response: Following your recommendation below, we now plan to conduct an expert Delphi study containing multiple Delphi rounds. We expect that experts will invest around 30 to 90 minutes per Delphi round depending on the underpinning aim of the Delphi round as well as experts’ familiarity and experiences with the topic. Participants will not receive an incentive for participation.

Reviewer comment: Why ‘simply’ a survey rather than a Delphi (or modified) approach? Will the survey (or Delphi) be pilot tested for question and response option clarity and language?

Authors’ response: This is an important point and we thank the reviewer for raising it. We agree that a Delphi study containing several Delphi rounds is suitable to collect the perspectives of international experts on citation tracking, promote discussions on the topic as well as to derive consensus recommendations for future practice and research on the use of citation tracking in systematic literature searching for health-related topics. We subjected our protocol to a major revision based on this point and now outline the methods for the Delphi. As stated in the revised text, we will pilot test and discuss our Delphi items with a person experienced in literature searching but who is not an author and not involved in the Delphi study.

Reviewer comment: The data analysis describes cross tables. I assume the authors mean cross tabulations which can be more than descriptive. Will the authors be conducting Chi-square analysis or other analytical approaches of the cross tabulations (e.g., p values)?

Authors’ response: We revised the corresponding section to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: The ethical concerns section is likely jurisdiction specific. In my setting ethics would be required. Can the authors explicitly indicate whether ethics is required or not. The section is currently vague on this critical issue.

Authors’ response: We added details to the ethical concerns section. With regard to the Swiss Human Research Act, our research does not concern human diseases and the structure and function of the human body. We will therefore not apply for ethical approval of the expert Delphi study.

Reviewer comment: For the survey, do the authors have an estimated sample size they are aiming for. Similarly, do the authors have an estimated response rate they are aiming for?

Authors’ response: This answer concerns the recruitment of experts for the Delphi study, not the survey. We intend to recruit at least 15 participants by using our stepwise approach for recruitment, the person-based approach (contacting authors of pertinent articles identified during the literature search as well as experts from authors’ professional networks) and the organisation-based approach (contacting national and international organisations and systematic review collaborations).
Competing Interests: None Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 04 Aug 2021

Julian Hirt, Institute of Applied Nursing Science, Department of Health, Eastern Switzerland University of Applied Sciences (formerly FHS St.Gallen), St.Gallen, Switzerland

04 Aug 2021

Author Response

Dear David Moher

We are grateful for your reviewer report and, in particular, for your helpful suggestions to derive evidence-based recommendations from our study. Below, we provide a detailed ... Continue reading Dear David Moher

We are grateful for your reviewer report and, in particular, for your helpful suggestions to derive evidence-based recommendations from our study. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
You will see that in response to your comments, we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope that these changes fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: A more conceptual question is to what degree this initiative overlaps and differs from PRISMA-S (Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. doi: 10.1186/s13643-020-01542-z).

Authors’ response: PRISMA-S is a reporting guideline for systematic literature searches. However, it is lacking on precise recommendations how to report citation tracking methods and results. By conducting a Delphi study with selected experts in the field (see our answer below for details), we aim at developing more detailed recommendations on adequately reporting citation tracking methods as part of systematic literature searching. This may complement recommendations on reporting given by the PRISMA-S group or could even be integrated in future versions. For the Delphi, we will invite the PRISMA-S group to best use synergies and experiences.

Reviewer comment: For example, item 5 of PRIMSA-S – Citation searching - states “Citation searching 5 Indicate whether cited references or citing references were examined, and describe any methods used for locating cited/citing references (e.g., browsing reference lists, using a citation index, setting up email alerts for references citing included studies).” I think this is important so as not to potentially confuse readers. Indeed, should the current initiative be seen as an implementation of PRISMA-S?

Authors’ response: See answer above. An important detail for the reporting of our study: since we will use citation tracking as part of our information retrieval, we will use PRISMA-S to guide our reporting of the scoping review. In the revised version, we reference it accordingly: “We will document our search strategy according to PRISMA-S”.

Reviewer comment: I had more difficulty understanding the proposed survey. For me, much of the methods were missing. I understand the recruitment (well reported). I would recommend the authors include the PRISMA-S group unless there is a lot of overlap between PRISMA-S and the other groups mentioned.

Authors’ response: As detailed further below, the survey will be replaced by a Delphi study. So, we replaced the survey methods with more detailed methods for the Delphi study. Concerning PRISMA-S, we agree with the reviewer and will ask the PRISMA-S working group to participate in our Delphi study (see below).

Reviewer comment: It was not clear to me how long the authors will make completing the survey– 15 minutes of 45 minutes? Similarly, will the respondents receive an incentive for completing the survey?

Authors’ response: Following your recommendation below, we now plan to conduct an expert Delphi study containing multiple Delphi rounds. We expect that experts will invest around 30 to 90 minutes per Delphi round depending on the underpinning aim of the Delphi round as well as experts’ familiarity and experiences with the topic. Participants will not receive an incentive for participation.

Reviewer comment: Why ‘simply’ a survey rather than a Delphi (or modified) approach? Will the survey (or Delphi) be pilot tested for question and response option clarity and language?

Authors’ response: This is an important point and we thank the reviewer for raising it. We agree that a Delphi study containing several Delphi rounds is suitable to collect the perspectives of international experts on citation tracking, promote discussions on the topic as well as to derive consensus recommendations for future practice and research on the use of citation tracking in systematic literature searching for health-related topics. We subjected our protocol to a major revision based on this point and now outline the methods for the Delphi. As stated in the revised text, we will pilot test and discuss our Delphi items with a person experienced in literature searching but who is not an author and not involved in the Delphi study.

Reviewer comment: The data analysis describes cross tables. I assume the authors mean cross tabulations which can be more than descriptive. Will the authors be conducting Chi-square analysis or other analytical approaches of the cross tabulations (e.g., p values)?

Authors’ response: We revised the corresponding section to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: The ethical concerns section is likely jurisdiction specific. In my setting ethics would be required. Can the authors explicitly indicate whether ethics is required or not. The section is currently vague on this critical issue.

Authors’ response: We added details to the ethical concerns section. With regard to the Swiss Human Research Act, our research does not concern human diseases and the structure and function of the human body. We will therefore not apply for ethical approval of the expert Delphi study.

Reviewer comment: For the survey, do the authors have an estimated sample size they are aiming for. Similarly, do the authors have an estimated response rate they are aiming for?

Authors’ response: This answer concerns the recruitment of experts for the Delphi study, not the survey. We intend to recruit at least 15 participants by using our stepwise approach for recruitment, the person-based approach (contacting authors of pertinent articles identified during the literature search as well as experts from authors’ professional networks) and the organisation-based approach (contacting national and international organisations and systematic review collaborations).
Dear David Moher

We are grateful for your reviewer report and, in particular, for your helpful suggestions to derive evidence-based recommendations from our study. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
You will see that in response to your comments, we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope that these changes fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: A more conceptual question is to what degree this initiative overlaps and differs from PRISMA-S (Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. doi: 10.1186/s13643-020-01542-z).

Authors’ response: PRISMA-S is a reporting guideline for systematic literature searches. However, it is lacking on precise recommendations how to report citation tracking methods and results. By conducting a Delphi study with selected experts in the field (see our answer below for details), we aim at developing more detailed recommendations on adequately reporting citation tracking methods as part of systematic literature searching. This may complement recommendations on reporting given by the PRISMA-S group or could even be integrated in future versions. For the Delphi, we will invite the PRISMA-S group to best use synergies and experiences.

Reviewer comment: For example, item 5 of PRIMSA-S – Citation searching - states “Citation searching 5 Indicate whether cited references or citing references were examined, and describe any methods used for locating cited/citing references (e.g., browsing reference lists, using a citation index, setting up email alerts for references citing included studies).” I think this is important so as not to potentially confuse readers. Indeed, should the current initiative be seen as an implementation of PRISMA-S?

Authors’ response: See answer above. An important detail for the reporting of our study: since we will use citation tracking as part of our information retrieval, we will use PRISMA-S to guide our reporting of the scoping review. In the revised version, we reference it accordingly: “We will document our search strategy according to PRISMA-S”.

Reviewer comment: I had more difficulty understanding the proposed survey. For me, much of the methods were missing. I understand the recruitment (well reported). I would recommend the authors include the PRISMA-S group unless there is a lot of overlap between PRISMA-S and the other groups mentioned.

Authors’ response: As detailed further below, the survey will be replaced by a Delphi study. So, we replaced the survey methods with more detailed methods for the Delphi study. Concerning PRISMA-S, we agree with the reviewer and will ask the PRISMA-S working group to participate in our Delphi study (see below).

Reviewer comment: It was not clear to me how long the authors will make completing the survey– 15 minutes of 45 minutes? Similarly, will the respondents receive an incentive for completing the survey?

Authors’ response: Following your recommendation below, we now plan to conduct an expert Delphi study containing multiple Delphi rounds. We expect that experts will invest around 30 to 90 minutes per Delphi round depending on the underpinning aim of the Delphi round as well as experts’ familiarity and experiences with the topic. Participants will not receive an incentive for participation.

Reviewer comment: Why ‘simply’ a survey rather than a Delphi (or modified) approach? Will the survey (or Delphi) be pilot tested for question and response option clarity and language?

Authors’ response: This is an important point and we thank the reviewer for raising it. We agree that a Delphi study containing several Delphi rounds is suitable to collect the perspectives of international experts on citation tracking, promote discussions on the topic as well as to derive consensus recommendations for future practice and research on the use of citation tracking in systematic literature searching for health-related topics. We subjected our protocol to a major revision based on this point and now outline the methods for the Delphi. As stated in the revised text, we will pilot test and discuss our Delphi items with a person experienced in literature searching but who is not an author and not involved in the Delphi study.

Reviewer comment: The data analysis describes cross tables. I assume the authors mean cross tabulations which can be more than descriptive. Will the authors be conducting Chi-square analysis or other analytical approaches of the cross tabulations (e.g., p values)?

Authors’ response: We revised the corresponding section to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: The ethical concerns section is likely jurisdiction specific. In my setting ethics would be required. Can the authors explicitly indicate whether ethics is required or not. The section is currently vague on this critical issue.

Authors’ response: We added details to the ethical concerns section. With regard to the Swiss Human Research Act, our research does not concern human diseases and the structure and function of the human body. We will therefore not apply for ethical approval of the expert Delphi study.

Reviewer comment: For the survey, do the authors have an estimated sample size they are aiming for. Similarly, do the authors have an estimated response rate they are aiming for?

Authors’ response: This answer concerns the recruitment of experts for the Delphi study, not the survey. We intend to recruit at least 15 participants by using our stepwise approach for recruitment, the person-based approach (contacting authors of pertinent articles identified during the literature search as well as experts from authors’ professional networks) and the organisation-based approach (contacting national and international organisations and systematic review collaborations).
Competing Interests: None Close
Report a concern

Views

Reviewer Report 05 Jan 2021

Julie Glanville, Cochrane Information Retrieval Methods Group, London, UK

Approved with Reservations

https://doi.org/10.5256/f1000research.30208.r76282

The authors propose to conduct a scoping review of the use of citation tracking techniques to identify research evidence to inform systematic reviews and also to survey experts on their use of citation tracking.

My suggestions for ... Continue reading

Introduction: 'These references are usually eligible for inclusion into the review and known at the start of the citation search10–12.' I think this could be reworded for clarity along the following lines:
'These references are usually eligible for inclusion into the review and some may be known at the beginning of the review and others may emerge as eligible records following the main database searches10–12.'
Then perhaps you may not need the sentence that follows the one I am commenting on.
Introduction: 'The taxonomy used to describe the…'. I think it might be better to use a word such as 'terminology' rather than 'taxonomy'.
Figure 1. The figure would be more helpful if it could show the relationships of the indirect citation tracking in the Figure 1b perhaps by colour coding. Please check diagrams in the following references which incorporate chronology into the citation picture as well as showing the indirect citation tracking clearly.
Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731-46.¹
Janssens A, Gwinn M, Brockman JE, Powell K, Goodman M. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol. 2020;20(1):25.²
Introduction: 'However, recent guidance suggests that combining several citation tracking methods (e.g. screening cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing. ' Please cite the guidance.
Introduction: ' It rather depends on a variety of factors. For instance, citation tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)'. These are all important issues and each deserves a bit more description so that readers can understand the differences. Again I think the word 'taxonomy' should be replaced with 'terminology'.
Introduction: current 'topical reviews' - suggest replacing with 'recent reviews on this topic'. Then please add in more citations so that 'reviews' can be supported with more than one reference.
Introduction: 'health-related systematic literature searching'. This occurs three times and for each occurrence I suggest rewording as follows for clarity: 'literature searching for systematic reviews of health and health-related topics'.
Eligibility criteria. 'evidence retrieval method'. Suggest that 'method' is not needed as the sentence has 'means'.
'Eligible studies need to have a health-related context. Studies without an explicitly specified research context are also eligible. ' This seems contradictory. Why are studies in education topics for example not useful if they are looking at the methods rather than the topic? Just including papers where the topic is not clear seems arbitrary when there may be much to learn from a paper even if it has an engineering topic, say. Perhaps reword along the lines of 'although studies undertaken in the context of health-related topics are the main focus, studies of citation tracking in other literatures will also be eligible where the focus is on exploring the methods rather than the subject topic'.
Table 1. Publication Type cannot be 'Any' when there are publication types listed as ineligible. The authors should list eligible study types or use some sort of exception wording.
Information sources. I am not sure 'free web searching via Google Scholar' is a helpful description, perhaps just say 'searches of Google Scholar' here and further down where it is mentioned again.
'For citation tracking, we will use Scopus, as this database seems to cover the largest number of relevant citations for the purpose of our review30. ' There is quite a lot of research that reports that other resources such as Google Scholar or Microsoft Academic provide wider coverage - the authors may wish to consider searching some of the free resources as well (since Scopus alone is not going to find the largest number of results) or finding other reasons for the use of a Scopus only approach.
'MEDBIB-L' - I think there may be a typo here and it is MEDLIB-L? I think it may also need to be corrected in the Dissemination section.
My understanding of the purpose of a protocol is that it should list what will happen and therefore the resources to be searched should be listed and there should be no examples which imply that other things may be added. So there should be no' e.g.'.
'text mining approach' - the chapter describes many approaches - which one was used in this case?
'parts of our textwords' - suggest 'some of our textwords'?
Box 1 - there is harmless redundancy in the search. For example co-citation is searched alone and in combination with other terms - it only needs to be searched alone.
Box 1 - with the focus on searching in the titles, the use of such close adjacency does not seem warranted - it would be more sensitive to use AND for title searching.
Box 1 - it might be helpful in the absence of subject headings and the focus on searching mainly in titles to also search the author keywords field.
Data charting process: I realise it is difficult to know what will happen but the authors description is not very clear about their plans 'We aim for an iterative data extraction process, but in the final publication, we will provide a detailed overview of extracted data items.' What is an 'iterative data extraction process' - what might it look like and what does it involve?
Data collection 'preferred taxonomy' - again, I suggest the word you mean is 'terminology' not 'taxonomy'.
Data analysis. 'To analyse the survey data, we will apply descriptive statistics based on frequencies, percentages, and cross tables.' I think it may be 'cross tabulations' rather than 'cross tables'?
Dissemination of results - it might be useful to also to try to present results at conferences of systematic review experts - Cochrane Colloquium and the Health Technology Assessment International (HTAi) conferences. These organisations also have active information retrieval special interest groups.
Conclusions: 'Depending on the available study landscape' - this seems rather vague to me - it would help the reader if the authors could be more explicit about what they mean.
'consequently, on the quality of clinical care' - I think it might not only be clinical care that could be impacted - it might be any aspect of health care depending on what is being reviewed e.g. a service, a policy, a new method of organising staff etc.
'in a health-related context may prove relevant also for other academic fields such as social or environmental sciences' - I think that much of what you find may be generalisable to other disciplines, so I suggest that you state this earlier as a possibility.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

References

1. Belter C: A relevance ranking method for citation-based search results. Scientometrics. 2017; 112 (2): 731-746 Publisher Full Text
2. Janssens A, Gwinn M, Brockman J, Powell K, et al.: Novel citation-based search method for scientific literature: a validation study. BMC Medical Research Methodology. 2020; 20 (1). Publisher Full Text

Competing Interests: I conduct training courses in information retrieval for the purpose of systematic reviews and these include courses in citation tracking/citation searches.

Reviewer Expertise: Information retrieval for evidence identification for systematic reviews.

CITE

Report a concern

Author Response 04 Aug 2021

Julian Hirt, Institute of Applied Nursing Science, Department of Health, Eastern Switzerland University of Applied Sciences (formerly FHS St.Gallen), St.Gallen, Switzerland

04 Aug 2021

Author Response
Dear Julie Glanville

We are grateful for your reviewer report and, in particular, for your helpful suggestions for improvement of our scoping review protocol. Below, we provide a detailed ... Continue reading
Dear Julie Glanville

We are grateful for your reviewer report and, in particular, for your helpful suggestions for improvement of our scoping review protocol. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
Please note that in response to the comments by reviewer 2 (David Moher), we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope our revisions now fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: Introduction: 'These references are usually eligible for inclusion into the review and known at the start of the citation search10–12.' I think this could be reworded for clarity along the following lines:
'These references are usually eligible for inclusion into the review and some may be known at the beginning of the review and others may emerge as eligible records following the main database searches10–12.'
Then perhaps you may not need the sentence that follows the one I am commenting on.

Authors’ response: We adapted this sentence and the subsequent one according to your suggestion.

Reviewer comment: Introduction: 'The taxonomy used to describe the…'. I think it might be better to use a word such as 'terminology' rather than 'taxonomy'.

Authors’ response: Thank you. We now use terminology instead of taxonomy throughout the manuscript.

Reviewer comment: Figure 1. The figure would be more helpful if it could show the relationships of the indirect citation tracking in the Figure 1b perhaps by colour coding. Please check diagrams in the following references which incorporate chronology into the citation picture as well as showing the indirect citation tracking clearly.
Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731-46.1
Janssens A, Gwinn M, Brockman JE, Powell K, Goodman M. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol. 2020;20(1):25.2

Authors’ response: We have colour coded the 4 citation tracking techniques as suggested. We had already consulted the citation tracking figures of Belter et al. and Janssens. et al. amongst others. Furthermore, we have added text to the figure caption. We feel that our figure is now clearer with respect to chronology.

Reviewer comment: Introduction: 'However, recent guidance suggests that combining several citation tracking methods (e.g. screening cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing.' Please cite the guidance.

Authors’ response: We added a reference.

Reviewer comment: Introduction: ' It rather depends on a variety of factors. For instance, citation tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)'. These are all important issues and each deserves a bit more description so that readers can understand the differences. Again I think the word 'taxonomy' should be replaced with 'terminology'.

Authors’ response: Thank you for this point! We have revised and elaborated this section.

Reviewer comment: Introduction: current 'topical reviews' - suggest replacing with 'recent reviews on this topic'. Then please add in more citations so that 'reviews' can be supported with more than one reference.

Authors’ response: We adapted the sentence according to your suggestion, thank you.

Reviewer comment: Introduction: 'health-related systematic literature searching'. This occurs three times and for each occurrence I suggest rewording as follows for clarity: 'literature searching for systematic reviews of health and health-related topics'.

Authors’ response: Thank you for your proposition. We now use “systematic literature searching for health-related topics“.

Reviewer comment: Eligibility criteria. 'evidence retrieval method'. Suggest that 'method' is not needed as the sentence has 'means'.

Authors’ response: We deleted the word ‘method’.

Reviewer comment: 'Eligible studies need to have a health-related context. Studies without an explicitly specified research context are also eligible.' This seems contradictory. Why are studies in education topics for example not useful if they are looking at the methods rather than the topic? Just including papers where the topic is not clear seems arbitrary when there may be much to learn from a paper even if it has an engineering topic, say. Perhaps reword along the lines of 'although studies undertaken in the context of health-related topics are the main focus, studies of citation tracking in other literatures will also be eligible where the focus is on exploring the methods rather than the subject topic'.

Authors’ response: We agree that it is interesting to learn about citation tracking from other disciplines. However, we have decided to focus on studies in a health-related context because this enables us to (i) consider the context in which the studies were conducted and to (ii) specifically direct our conclusions and recommendations to the health context. Furthermore, we concentrate on health-related studies for practical reasons facing the high number of references that need screening. We deleted the sentence ‘Studies without an explicitly specified research context are also eligible’ to avoid misunderstandings.

On that note we also improved our exclusion criteria by adding this criterion: ‘any study only assessing the benefit of combined search methods (whereas the isolated benefit of citation tracking cannot be extracted)’.

Reviewer comment: Table 1. Publication Type cannot be 'Any' when there are publication types listed as ineligible. The authors should list eligible study types or use some sort of exception wording.

Authors’ response: We now use ‘Any reports of empirical studies’.

Reviewer comment: Information sources. I am not sure 'free web searching via Google Scholar' is a helpful description, perhaps just say 'searches of Google Scholar' here and further down where it is mentioned again.

Authors’ response: We now use ‘web searching’ instead of ‘free web searching’ throughout the manuscript.

Reviewer comment: 'For citation tracking, we will use Scopus, as this database seems to cover the largest number of relevant citations for the purpose of our review30.' There is quite a lot of research that reports that other resources such as Google Scholar or Microsoft Academic provide wider coverage - the authors may wish to consider searching some of the free resources as well (since Scopus alone is not going to find the largest number of results) or finding other reasons for the use of a Scopus only approach.

Authors’ response: We agree and now plan forward citation tracking using a triple approach: Scopus, Web of Science, and Google Scholar. We will iteratively repeat citation tracking on newly identified records to include until no further eligible references will be identified. We now describe this new approach in the methods.

Reviewer comment: 'MEDBIB-L' - I think there may be a typo here and it is MEDLIB-L? I think it may also need to be corrected in the Dissemination section.

Authors’ response: The correct name is MEDIBIB-L (the “I” was missing), a mailing list specific to libraries of medicine in the German-speaking area (library = Bibliothek; https://lists.uni-due.de/mailman/listinfo/medbib-l).

Reviewer comment: My understanding of the purpose of a protocol is that it should list what will happen and therefore the resources to be searched should be listed and there should be no examples which imply that other things may be added. So there should be no' e.g.'.

Authors’ response: We deleted ‘e.g.’

Reviewer comment: 'text mining approach' - the chapter describes many approaches - which one was used in this case?

Authors’ response: We used several tools that were used to improve our search. We now list all the tools that we used during search drafting phase in our manuscript and here:

PubMed PubReMiner: https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi

AntConc 3.5.7 (Windows), developed by Laurence Anthony, Faculty of Science and Engineering, Waseda University, Japan

Yale MeSH Analyzer: http://mesh.med.yale.edu/

TerMine: http://www.nactem.ac.uk/software/termine/

Text Analyzer: https://www.online-utility.org/text/analyzer.jsp

Voyant: https://voyant-tools.org/

VOSviewer: https://www.vosviewer.comrt

Reviewer comment: 'parts of our textwords' - suggest 'some of our textwords'?

Authors’ response: We agree, thank you.

Reviewer comment: Box 1 - there is harmless redundancy in the search. For example co-citation is searched alone and in combination with other terms - it only needs to be searched alone.

Authors’ response: Yes, these are remnants of a previously more specific search and indeed now redundant. Since we already ran the searches by now, we would prefer to leave the search string exactly as we ran it for precise documentation. However, we noted this in our search files in case of a later update.

Reviewer comment: Box 1 - with the focus on searching in the titles, the use of such close adjacency does not seem warranted - it would be more sensitive to use AND for title searching.

Authors’ response: Again, this is very good input which we noted in our search files in case of a later update. Unfortunately, we already ran the searches by now and are done screening. We are, however, confident that any article that we missed due to the close adjacency will be retrieved through our extensive citation tracking, web searches or expert contacting.

Reviewer comment: Box 1 - it might be helpful in the absence of subject headings and the focus on searching mainly in titles to also search the author keywords field.

Authors’ response: Thank you for the sensible suggestion which we also noted in our search files in case of a later update.

Reviewer comment: Data charting process: I realise it is difficult to know what will happen but the authors description is not very clear about their plans 'We aim for an iterative data extraction process, but in the final publication, we will provide a detailed overview of extracted data items.' What is an 'iterative data extraction process' - what might it look like and what does it involve?

Authors’ response: We revised this paragraph and provide more details: ‘Since we expect heterogeneous studies in terms of aim, design, and methods, we aim for an iterative data extraction process. This allows a flexible and study-specific data extraction process, e.g. by adding previously neglected data extraction items that might contribute to the overall body of knowledge to the data extraction form.’

Reviewer comment: Data collection 'preferred taxonomy' - again, I suggest the word you mean is 'terminology' not 'taxonomy'.

Authors’ response: Indeed, thank you.

Reviewer comment: Data analysis. 'To analyse the survey data, we will apply descriptive statistics based on frequencies, percentages, and cross tables.' I think it may be 'cross tabulations' rather than 'cross tables'?

Authors’ response: We revised this paragraph to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: Dissemination of results - it might be useful to also to try to present results at conferences of systematic review experts - Cochrane Colloquium and the Health Technology Assessment International (HTAi) conferences. These organisations also have active information retrieval special interest groups.

Authors’ response: Thank you, we added these conferences to our list of potential conferences.

Reviewer comment: Conclusions: 'Depending on the available study landscape' - this seems rather vague to me - it would help the reader if the authors could be more explicit about what they mean.

Authors’ response: We deleted ‘Depending on the available study landscape’ to be more precise.

Reviewer comment: 'consequently, on the quality of clinical care' - I think it might not only be clinical care that could be impacted - it might be any aspect of health care depending on what is being reviewed e.g. a service, a policy, a new method of organising staff etc.

Authors’ response: We agree and revised the sentence to ‘health care’ instead of ‘clinical care’.

Reviewer comment: 'in a health-related context may prove relevant also for other academic fields such as social or environmental sciences' - I think that much of what you find may be generalisable to other disciplines, so I suggest that you state this earlier as a possibility.

Authors’ response: We agree that our results may indeed prove relevant to other disciplines. However, we decided to narrow our research efforts to the health-related fields, so we think this should be rather a part of the discussion/outlook than earlier in the manuscript. We changed the sentence to underline that aspect: “Although we solely focus on a health-related context, it is possible that some of the recommendations developed during this project may prove relevant also for other academic fields such as social or environmental sciences”.
Dear Julie Glanville

We are grateful for your reviewer report and, in particular, for your helpful suggestions for improvement of our scoping review protocol. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
Please note that in response to the comments by reviewer 2 (David Moher), we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope our revisions now fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: Introduction: 'These references are usually eligible for inclusion into the review and known at the start of the citation search10–12.' I think this could be reworded for clarity along the following lines:
'These references are usually eligible for inclusion into the review and some may be known at the beginning of the review and others may emerge as eligible records following the main database searches10–12.'
Then perhaps you may not need the sentence that follows the one I am commenting on.

Authors’ response: We adapted this sentence and the subsequent one according to your suggestion.

Reviewer comment: Introduction: 'The taxonomy used to describe the…'. I think it might be better to use a word such as 'terminology' rather than 'taxonomy'.

Authors’ response: Thank you. We now use terminology instead of taxonomy throughout the manuscript.

Reviewer comment: Figure 1. The figure would be more helpful if it could show the relationships of the indirect citation tracking in the Figure 1b perhaps by colour coding. Please check diagrams in the following references which incorporate chronology into the citation picture as well as showing the indirect citation tracking clearly.
Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731-46.1
Janssens A, Gwinn M, Brockman JE, Powell K, Goodman M. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol. 2020;20(1):25.2

Authors’ response: We have colour coded the 4 citation tracking techniques as suggested. We had already consulted the citation tracking figures of Belter et al. and Janssens. et al. amongst others. Furthermore, we have added text to the figure caption. We feel that our figure is now clearer with respect to chronology.

Reviewer comment: Introduction: 'However, recent guidance suggests that combining several citation tracking methods (e.g. screening cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing.' Please cite the guidance.

Authors’ response: We added a reference.

Reviewer comment: Introduction: ' It rather depends on a variety of factors. For instance, citation tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)'. These are all important issues and each deserves a bit more description so that readers can understand the differences. Again I think the word 'taxonomy' should be replaced with 'terminology'.

Authors’ response: Thank you for this point! We have revised and elaborated this section.

Reviewer comment: Introduction: current 'topical reviews' - suggest replacing with 'recent reviews on this topic'. Then please add in more citations so that 'reviews' can be supported with more than one reference.

Authors’ response: We adapted the sentence according to your suggestion, thank you.

Reviewer comment: Introduction: 'health-related systematic literature searching'. This occurs three times and for each occurrence I suggest rewording as follows for clarity: 'literature searching for systematic reviews of health and health-related topics'.

Authors’ response: Thank you for your proposition. We now use “systematic literature searching for health-related topics“.

Reviewer comment: Eligibility criteria. 'evidence retrieval method'. Suggest that 'method' is not needed as the sentence has 'means'.

Authors’ response: We deleted the word ‘method’.

Reviewer comment: 'Eligible studies need to have a health-related context. Studies without an explicitly specified research context are also eligible.' This seems contradictory. Why are studies in education topics for example not useful if they are looking at the methods rather than the topic? Just including papers where the topic is not clear seems arbitrary when there may be much to learn from a paper even if it has an engineering topic, say. Perhaps reword along the lines of 'although studies undertaken in the context of health-related topics are the main focus, studies of citation tracking in other literatures will also be eligible where the focus is on exploring the methods rather than the subject topic'.

Authors’ response: We agree that it is interesting to learn about citation tracking from other disciplines. However, we have decided to focus on studies in a health-related context because this enables us to (i) consider the context in which the studies were conducted and to (ii) specifically direct our conclusions and recommendations to the health context. Furthermore, we concentrate on health-related studies for practical reasons facing the high number of references that need screening. We deleted the sentence ‘Studies without an explicitly specified research context are also eligible’ to avoid misunderstandings.

On that note we also improved our exclusion criteria by adding this criterion: ‘any study only assessing the benefit of combined search methods (whereas the isolated benefit of citation tracking cannot be extracted)’.

Reviewer comment: Table 1. Publication Type cannot be 'Any' when there are publication types listed as ineligible. The authors should list eligible study types or use some sort of exception wording.

Authors’ response: We now use ‘Any reports of empirical studies’.

Reviewer comment: Information sources. I am not sure 'free web searching via Google Scholar' is a helpful description, perhaps just say 'searches of Google Scholar' here and further down where it is mentioned again.

Authors’ response: We now use ‘web searching’ instead of ‘free web searching’ throughout the manuscript.

Reviewer comment: 'For citation tracking, we will use Scopus, as this database seems to cover the largest number of relevant citations for the purpose of our review30.' There is quite a lot of research that reports that other resources such as Google Scholar or Microsoft Academic provide wider coverage - the authors may wish to consider searching some of the free resources as well (since Scopus alone is not going to find the largest number of results) or finding other reasons for the use of a Scopus only approach.

Authors’ response: We agree and now plan forward citation tracking using a triple approach: Scopus, Web of Science, and Google Scholar. We will iteratively repeat citation tracking on newly identified records to include until no further eligible references will be identified. We now describe this new approach in the methods.

Reviewer comment: 'MEDBIB-L' - I think there may be a typo here and it is MEDLIB-L? I think it may also need to be corrected in the Dissemination section.

Authors’ response: The correct name is MEDIBIB-L (the “I” was missing), a mailing list specific to libraries of medicine in the German-speaking area (library = Bibliothek; https://lists.uni-due.de/mailman/listinfo/medbib-l).

Reviewer comment: My understanding of the purpose of a protocol is that it should list what will happen and therefore the resources to be searched should be listed and there should be no examples which imply that other things may be added. So there should be no' e.g.'.

Authors’ response: We deleted ‘e.g.’

Reviewer comment: 'text mining approach' - the chapter describes many approaches - which one was used in this case?

Authors’ response: We used several tools that were used to improve our search. We now list all the tools that we used during search drafting phase in our manuscript and here:

PubMed PubReMiner: https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi

AntConc 3.5.7 (Windows), developed by Laurence Anthony, Faculty of Science and Engineering, Waseda University, Japan

Yale MeSH Analyzer: http://mesh.med.yale.edu/

TerMine: http://www.nactem.ac.uk/software/termine/

Text Analyzer: https://www.online-utility.org/text/analyzer.jsp

Voyant: https://voyant-tools.org/

VOSviewer: https://www.vosviewer.comrt

Reviewer comment: 'parts of our textwords' - suggest 'some of our textwords'?

Authors’ response: We agree, thank you.

Reviewer comment: Box 1 - there is harmless redundancy in the search. For example co-citation is searched alone and in combination with other terms - it only needs to be searched alone.

Authors’ response: Yes, these are remnants of a previously more specific search and indeed now redundant. Since we already ran the searches by now, we would prefer to leave the search string exactly as we ran it for precise documentation. However, we noted this in our search files in case of a later update.

Reviewer comment: Box 1 - with the focus on searching in the titles, the use of such close adjacency does not seem warranted - it would be more sensitive to use AND for title searching.

Authors’ response: Again, this is very good input which we noted in our search files in case of a later update. Unfortunately, we already ran the searches by now and are done screening. We are, however, confident that any article that we missed due to the close adjacency will be retrieved through our extensive citation tracking, web searches or expert contacting.

Reviewer comment: Box 1 - it might be helpful in the absence of subject headings and the focus on searching mainly in titles to also search the author keywords field.

Authors’ response: Thank you for the sensible suggestion which we also noted in our search files in case of a later update.

Reviewer comment: Data charting process: I realise it is difficult to know what will happen but the authors description is not very clear about their plans 'We aim for an iterative data extraction process, but in the final publication, we will provide a detailed overview of extracted data items.' What is an 'iterative data extraction process' - what might it look like and what does it involve?

Authors’ response: We revised this paragraph and provide more details: ‘Since we expect heterogeneous studies in terms of aim, design, and methods, we aim for an iterative data extraction process. This allows a flexible and study-specific data extraction process, e.g. by adding previously neglected data extraction items that might contribute to the overall body of knowledge to the data extraction form.’

Reviewer comment: Data collection 'preferred taxonomy' - again, I suggest the word you mean is 'terminology' not 'taxonomy'.

Authors’ response: Indeed, thank you.

Reviewer comment: Data analysis. 'To analyse the survey data, we will apply descriptive statistics based on frequencies, percentages, and cross tables.' I think it may be 'cross tabulations' rather than 'cross tables'?

Authors’ response: We revised this paragraph to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: Dissemination of results - it might be useful to also to try to present results at conferences of systematic review experts - Cochrane Colloquium and the Health Technology Assessment International (HTAi) conferences. These organisations also have active information retrieval special interest groups.

Authors’ response: Thank you, we added these conferences to our list of potential conferences.

Reviewer comment: Conclusions: 'Depending on the available study landscape' - this seems rather vague to me - it would help the reader if the authors could be more explicit about what they mean.

Authors’ response: We deleted ‘Depending on the available study landscape’ to be more precise.

Reviewer comment: 'consequently, on the quality of clinical care' - I think it might not only be clinical care that could be impacted - it might be any aspect of health care depending on what is being reviewed e.g. a service, a policy, a new method of organising staff etc.

Authors’ response: We agree and revised the sentence to ‘health care’ instead of ‘clinical care’.

Reviewer comment: 'in a health-related context may prove relevant also for other academic fields such as social or environmental sciences' - I think that much of what you find may be generalisable to other disciplines, so I suggest that you state this earlier as a possibility.

Authors’ response: We agree that our results may indeed prove relevant to other disciplines. However, we decided to narrow our research efforts to the health-related fields, so we think this should be rather a part of the discussion/outlook than earlier in the manuscript. We changed the sentence to underline that aspect: “Although we solely focus on a health-related context, it is possible that some of the recommendations developed during this project may prove relevant also for other academic fields such as social or environmental sciences”.
Competing Interests: None Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 04 Aug 2021

Julian Hirt, Institute of Applied Nursing Science, Department of Health, Eastern Switzerland University of Applied Sciences (formerly FHS St.Gallen), St.Gallen, Switzerland

04 Aug 2021

Author Response
Dear Julie Glanville

We are grateful for your reviewer report and, in particular, for your helpful suggestions for improvement of our scoping review protocol. Below, we provide a detailed ... Continue reading
Dear Julie Glanville

We are grateful for your reviewer report and, in particular, for your helpful suggestions for improvement of our scoping review protocol. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
Please note that in response to the comments by reviewer 2 (David Moher), we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope our revisions now fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: Introduction: 'These references are usually eligible for inclusion into the review and known at the start of the citation search10–12.' I think this could be reworded for clarity along the following lines:
'These references are usually eligible for inclusion into the review and some may be known at the beginning of the review and others may emerge as eligible records following the main database searches10–12.'
Then perhaps you may not need the sentence that follows the one I am commenting on.

Authors’ response: We adapted this sentence and the subsequent one according to your suggestion.

Reviewer comment: Introduction: 'The taxonomy used to describe the…'. I think it might be better to use a word such as 'terminology' rather than 'taxonomy'.

Authors’ response: Thank you. We now use terminology instead of taxonomy throughout the manuscript.

Reviewer comment: Figure 1. The figure would be more helpful if it could show the relationships of the indirect citation tracking in the Figure 1b perhaps by colour coding. Please check diagrams in the following references which incorporate chronology into the citation picture as well as showing the indirect citation tracking clearly.
Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731-46.1
Janssens A, Gwinn M, Brockman JE, Powell K, Goodman M. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol. 2020;20(1):25.2

Authors’ response: We have colour coded the 4 citation tracking techniques as suggested. We had already consulted the citation tracking figures of Belter et al. and Janssens. et al. amongst others. Furthermore, we have added text to the figure caption. We feel that our figure is now clearer with respect to chronology.

Reviewer comment: Introduction: 'However, recent guidance suggests that combining several citation tracking methods (e.g. screening cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing.' Please cite the guidance.

Authors’ response: We added a reference.

Reviewer comment: Introduction: ' It rather depends on a variety of factors. For instance, citation tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)'. These are all important issues and each deserves a bit more description so that readers can understand the differences. Again I think the word 'taxonomy' should be replaced with 'terminology'.

Authors’ response: Thank you for this point! We have revised and elaborated this section.

Reviewer comment: Introduction: current 'topical reviews' - suggest replacing with 'recent reviews on this topic'. Then please add in more citations so that 'reviews' can be supported with more than one reference.

Authors’ response: We adapted the sentence according to your suggestion, thank you.

Reviewer comment: Introduction: 'health-related systematic literature searching'. This occurs three times and for each occurrence I suggest rewording as follows for clarity: 'literature searching for systematic reviews of health and health-related topics'.

Authors’ response: Thank you for your proposition. We now use “systematic literature searching for health-related topics“.

Reviewer comment: Eligibility criteria. 'evidence retrieval method'. Suggest that 'method' is not needed as the sentence has 'means'.

Authors’ response: We deleted the word ‘method’.

Reviewer comment: 'Eligible studies need to have a health-related context. Studies without an explicitly specified research context are also eligible.' This seems contradictory. Why are studies in education topics for example not useful if they are looking at the methods rather than the topic? Just including papers where the topic is not clear seems arbitrary when there may be much to learn from a paper even if it has an engineering topic, say. Perhaps reword along the lines of 'although studies undertaken in the context of health-related topics are the main focus, studies of citation tracking in other literatures will also be eligible where the focus is on exploring the methods rather than the subject topic'.

Authors’ response: We agree that it is interesting to learn about citation tracking from other disciplines. However, we have decided to focus on studies in a health-related context because this enables us to (i) consider the context in which the studies were conducted and to (ii) specifically direct our conclusions and recommendations to the health context. Furthermore, we concentrate on health-related studies for practical reasons facing the high number of references that need screening. We deleted the sentence ‘Studies without an explicitly specified research context are also eligible’ to avoid misunderstandings.

On that note we also improved our exclusion criteria by adding this criterion: ‘any study only assessing the benefit of combined search methods (whereas the isolated benefit of citation tracking cannot be extracted)’.

Reviewer comment: Table 1. Publication Type cannot be 'Any' when there are publication types listed as ineligible. The authors should list eligible study types or use some sort of exception wording.

Authors’ response: We now use ‘Any reports of empirical studies’.

Reviewer comment: Information sources. I am not sure 'free web searching via Google Scholar' is a helpful description, perhaps just say 'searches of Google Scholar' here and further down where it is mentioned again.

Authors’ response: We now use ‘web searching’ instead of ‘free web searching’ throughout the manuscript.

Reviewer comment: 'For citation tracking, we will use Scopus, as this database seems to cover the largest number of relevant citations for the purpose of our review30.' There is quite a lot of research that reports that other resources such as Google Scholar or Microsoft Academic provide wider coverage - the authors may wish to consider searching some of the free resources as well (since Scopus alone is not going to find the largest number of results) or finding other reasons for the use of a Scopus only approach.

Authors’ response: We agree and now plan forward citation tracking using a triple approach: Scopus, Web of Science, and Google Scholar. We will iteratively repeat citation tracking on newly identified records to include until no further eligible references will be identified. We now describe this new approach in the methods.

Reviewer comment: 'MEDBIB-L' - I think there may be a typo here and it is MEDLIB-L? I think it may also need to be corrected in the Dissemination section.

Authors’ response: The correct name is MEDIBIB-L (the “I” was missing), a mailing list specific to libraries of medicine in the German-speaking area (library = Bibliothek; https://lists.uni-due.de/mailman/listinfo/medbib-l).

Reviewer comment: My understanding of the purpose of a protocol is that it should list what will happen and therefore the resources to be searched should be listed and there should be no examples which imply that other things may be added. So there should be no' e.g.'.

Authors’ response: We deleted ‘e.g.’

Reviewer comment: 'text mining approach' - the chapter describes many approaches - which one was used in this case?

Authors’ response: We used several tools that were used to improve our search. We now list all the tools that we used during search drafting phase in our manuscript and here:

PubMed PubReMiner: https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi

AntConc 3.5.7 (Windows), developed by Laurence Anthony, Faculty of Science and Engineering, Waseda University, Japan

Yale MeSH Analyzer: http://mesh.med.yale.edu/

TerMine: http://www.nactem.ac.uk/software/termine/

Text Analyzer: https://www.online-utility.org/text/analyzer.jsp

Voyant: https://voyant-tools.org/

VOSviewer: https://www.vosviewer.comrt

Reviewer comment: 'parts of our textwords' - suggest 'some of our textwords'?

Authors’ response: We agree, thank you.

Reviewer comment: Box 1 - there is harmless redundancy in the search. For example co-citation is searched alone and in combination with other terms - it only needs to be searched alone.

Authors’ response: Yes, these are remnants of a previously more specific search and indeed now redundant. Since we already ran the searches by now, we would prefer to leave the search string exactly as we ran it for precise documentation. However, we noted this in our search files in case of a later update.

Reviewer comment: Box 1 - with the focus on searching in the titles, the use of such close adjacency does not seem warranted - it would be more sensitive to use AND for title searching.

Authors’ response: Again, this is very good input which we noted in our search files in case of a later update. Unfortunately, we already ran the searches by now and are done screening. We are, however, confident that any article that we missed due to the close adjacency will be retrieved through our extensive citation tracking, web searches or expert contacting.

Reviewer comment: Box 1 - it might be helpful in the absence of subject headings and the focus on searching mainly in titles to also search the author keywords field.

Authors’ response: Thank you for the sensible suggestion which we also noted in our search files in case of a later update.

Reviewer comment: Data charting process: I realise it is difficult to know what will happen but the authors description is not very clear about their plans 'We aim for an iterative data extraction process, but in the final publication, we will provide a detailed overview of extracted data items.' What is an 'iterative data extraction process' - what might it look like and what does it involve?

Authors’ response: We revised this paragraph and provide more details: ‘Since we expect heterogeneous studies in terms of aim, design, and methods, we aim for an iterative data extraction process. This allows a flexible and study-specific data extraction process, e.g. by adding previously neglected data extraction items that might contribute to the overall body of knowledge to the data extraction form.’

Reviewer comment: Data collection 'preferred taxonomy' - again, I suggest the word you mean is 'terminology' not 'taxonomy'.

Authors’ response: Indeed, thank you.

Reviewer comment: Data analysis. 'To analyse the survey data, we will apply descriptive statistics based on frequencies, percentages, and cross tables.' I think it may be 'cross tabulations' rather than 'cross tables'?

Authors’ response: We revised this paragraph to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: Dissemination of results - it might be useful to also to try to present results at conferences of systematic review experts - Cochrane Colloquium and the Health Technology Assessment International (HTAi) conferences. These organisations also have active information retrieval special interest groups.

Authors’ response: Thank you, we added these conferences to our list of potential conferences.

Reviewer comment: Conclusions: 'Depending on the available study landscape' - this seems rather vague to me - it would help the reader if the authors could be more explicit about what they mean.

Authors’ response: We deleted ‘Depending on the available study landscape’ to be more precise.

Reviewer comment: 'consequently, on the quality of clinical care' - I think it might not only be clinical care that could be impacted - it might be any aspect of health care depending on what is being reviewed e.g. a service, a policy, a new method of organising staff etc.

Authors’ response: We agree and revised the sentence to ‘health care’ instead of ‘clinical care’.

Reviewer comment: 'in a health-related context may prove relevant also for other academic fields such as social or environmental sciences' - I think that much of what you find may be generalisable to other disciplines, so I suggest that you state this earlier as a possibility.

Authors’ response: We agree that our results may indeed prove relevant to other disciplines. However, we decided to narrow our research efforts to the health-related fields, so we think this should be rather a part of the discussion/outlook than earlier in the manuscript. We changed the sentence to underline that aspect: “Although we solely focus on a health-related context, it is possible that some of the recommendations developed during this project may prove relevant also for other academic fields such as social or environmental sciences”.
Dear Julie Glanville

We are grateful for your reviewer report and, in particular, for your helpful suggestions for improvement of our scoping review protocol. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
Please note that in response to the comments by reviewer 2 (David Moher), we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope our revisions now fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: Introduction: 'These references are usually eligible for inclusion into the review and known at the start of the citation search10–12.' I think this could be reworded for clarity along the following lines:
'These references are usually eligible for inclusion into the review and some may be known at the beginning of the review and others may emerge as eligible records following the main database searches10–12.'
Then perhaps you may not need the sentence that follows the one I am commenting on.

Authors’ response: We adapted this sentence and the subsequent one according to your suggestion.

Reviewer comment: Introduction: 'The taxonomy used to describe the…'. I think it might be better to use a word such as 'terminology' rather than 'taxonomy'.

Authors’ response: Thank you. We now use terminology instead of taxonomy throughout the manuscript.

Reviewer comment: Figure 1. The figure would be more helpful if it could show the relationships of the indirect citation tracking in the Figure 1b perhaps by colour coding. Please check diagrams in the following references which incorporate chronology into the citation picture as well as showing the indirect citation tracking clearly.
Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731-46.1
Janssens A, Gwinn M, Brockman JE, Powell K, Goodman M. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol. 2020;20(1):25.2

Authors’ response: We have colour coded the 4 citation tracking techniques as suggested. We had already consulted the citation tracking figures of Belter et al. and Janssens. et al. amongst others. Furthermore, we have added text to the figure caption. We feel that our figure is now clearer with respect to chronology.

Reviewer comment: Introduction: 'However, recent guidance suggests that combining several citation tracking methods (e.g. screening cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing.' Please cite the guidance.

Authors’ response: We added a reference.

Reviewer comment: Introduction: ' It rather depends on a variety of factors. For instance, citation tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)'. These are all important issues and each deserves a bit more description so that readers can understand the differences. Again I think the word 'taxonomy' should be replaced with 'terminology'.

Authors’ response: Thank you for this point! We have revised and elaborated this section.

Reviewer comment: Introduction: current 'topical reviews' - suggest replacing with 'recent reviews on this topic'. Then please add in more citations so that 'reviews' can be supported with more than one reference.

Authors’ response: We adapted the sentence according to your suggestion, thank you.

Reviewer comment: Introduction: 'health-related systematic literature searching'. This occurs three times and for each occurrence I suggest rewording as follows for clarity: 'literature searching for systematic reviews of health and health-related topics'.

Authors’ response: Thank you for your proposition. We now use “systematic literature searching for health-related topics“.

Reviewer comment: Eligibility criteria. 'evidence retrieval method'. Suggest that 'method' is not needed as the sentence has 'means'.

Authors’ response: We deleted the word ‘method’.

Reviewer comment: 'Eligible studies need to have a health-related context. Studies without an explicitly specified research context are also eligible.' This seems contradictory. Why are studies in education topics for example not useful if they are looking at the methods rather than the topic? Just including papers where the topic is not clear seems arbitrary when there may be much to learn from a paper even if it has an engineering topic, say. Perhaps reword along the lines of 'although studies undertaken in the context of health-related topics are the main focus, studies of citation tracking in other literatures will also be eligible where the focus is on exploring the methods rather than the subject topic'.

Authors’ response: We agree that it is interesting to learn about citation tracking from other disciplines. However, we have decided to focus on studies in a health-related context because this enables us to (i) consider the context in which the studies were conducted and to (ii) specifically direct our conclusions and recommendations to the health context. Furthermore, we concentrate on health-related studies for practical reasons facing the high number of references that need screening. We deleted the sentence ‘Studies without an explicitly specified research context are also eligible’ to avoid misunderstandings.

On that note we also improved our exclusion criteria by adding this criterion: ‘any study only assessing the benefit of combined search methods (whereas the isolated benefit of citation tracking cannot be extracted)’.

Reviewer comment: Table 1. Publication Type cannot be 'Any' when there are publication types listed as ineligible. The authors should list eligible study types or use some sort of exception wording.

Authors’ response: We now use ‘Any reports of empirical studies’.

Reviewer comment: Information sources. I am not sure 'free web searching via Google Scholar' is a helpful description, perhaps just say 'searches of Google Scholar' here and further down where it is mentioned again.

Authors’ response: We now use ‘web searching’ instead of ‘free web searching’ throughout the manuscript.

Reviewer comment: 'For citation tracking, we will use Scopus, as this database seems to cover the largest number of relevant citations for the purpose of our review30.' There is quite a lot of research that reports that other resources such as Google Scholar or Microsoft Academic provide wider coverage - the authors may wish to consider searching some of the free resources as well (since Scopus alone is not going to find the largest number of results) or finding other reasons for the use of a Scopus only approach.

Authors’ response: We agree and now plan forward citation tracking using a triple approach: Scopus, Web of Science, and Google Scholar. We will iteratively repeat citation tracking on newly identified records to include until no further eligible references will be identified. We now describe this new approach in the methods.

Reviewer comment: 'MEDBIB-L' - I think there may be a typo here and it is MEDLIB-L? I think it may also need to be corrected in the Dissemination section.

Authors’ response: The correct name is MEDIBIB-L (the “I” was missing), a mailing list specific to libraries of medicine in the German-speaking area (library = Bibliothek; https://lists.uni-due.de/mailman/listinfo/medbib-l).

Reviewer comment: My understanding of the purpose of a protocol is that it should list what will happen and therefore the resources to be searched should be listed and there should be no examples which imply that other things may be added. So there should be no' e.g.'.

Authors’ response: We deleted ‘e.g.’

Reviewer comment: 'text mining approach' - the chapter describes many approaches - which one was used in this case?

Authors’ response: We used several tools that were used to improve our search. We now list all the tools that we used during search drafting phase in our manuscript and here:

PubMed PubReMiner: https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi

AntConc 3.5.7 (Windows), developed by Laurence Anthony, Faculty of Science and Engineering, Waseda University, Japan

Yale MeSH Analyzer: http://mesh.med.yale.edu/

TerMine: http://www.nactem.ac.uk/software/termine/

Text Analyzer: https://www.online-utility.org/text/analyzer.jsp

Voyant: https://voyant-tools.org/

VOSviewer: https://www.vosviewer.comrt

Reviewer comment: 'parts of our textwords' - suggest 'some of our textwords'?

Authors’ response: We agree, thank you.

Reviewer comment: Box 1 - there is harmless redundancy in the search. For example co-citation is searched alone and in combination with other terms - it only needs to be searched alone.

Authors’ response: Yes, these are remnants of a previously more specific search and indeed now redundant. Since we already ran the searches by now, we would prefer to leave the search string exactly as we ran it for precise documentation. However, we noted this in our search files in case of a later update.

Reviewer comment: Box 1 - with the focus on searching in the titles, the use of such close adjacency does not seem warranted - it would be more sensitive to use AND for title searching.

Authors’ response: Again, this is very good input which we noted in our search files in case of a later update. Unfortunately, we already ran the searches by now and are done screening. We are, however, confident that any article that we missed due to the close adjacency will be retrieved through our extensive citation tracking, web searches or expert contacting.

Reviewer comment: Box 1 - it might be helpful in the absence of subject headings and the focus on searching mainly in titles to also search the author keywords field.

Authors’ response: Thank you for the sensible suggestion which we also noted in our search files in case of a later update.

Reviewer comment: Data charting process: I realise it is difficult to know what will happen but the authors description is not very clear about their plans 'We aim for an iterative data extraction process, but in the final publication, we will provide a detailed overview of extracted data items.' What is an 'iterative data extraction process' - what might it look like and what does it involve?

Authors’ response: We revised this paragraph and provide more details: ‘Since we expect heterogeneous studies in terms of aim, design, and methods, we aim for an iterative data extraction process. This allows a flexible and study-specific data extraction process, e.g. by adding previously neglected data extraction items that might contribute to the overall body of knowledge to the data extraction form.’

Reviewer comment: Data collection 'preferred taxonomy' - again, I suggest the word you mean is 'terminology' not 'taxonomy'.

Authors’ response: Indeed, thank you.

Reviewer comment: Data analysis. 'To analyse the survey data, we will apply descriptive statistics based on frequencies, percentages, and cross tables.' I think it may be 'cross tabulations' rather than 'cross tables'?

Authors’ response: We revised this paragraph to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: Dissemination of results - it might be useful to also to try to present results at conferences of systematic review experts - Cochrane Colloquium and the Health Technology Assessment International (HTAi) conferences. These organisations also have active information retrieval special interest groups.

Authors’ response: Thank you, we added these conferences to our list of potential conferences.

Reviewer comment: Conclusions: 'Depending on the available study landscape' - this seems rather vague to me - it would help the reader if the authors could be more explicit about what they mean.

Authors’ response: We deleted ‘Depending on the available study landscape’ to be more precise.

Reviewer comment: 'consequently, on the quality of clinical care' - I think it might not only be clinical care that could be impacted - it might be any aspect of health care depending on what is being reviewed e.g. a service, a policy, a new method of organising staff etc.

Authors’ response: We agree and revised the sentence to ‘health care’ instead of ‘clinical care’.

Reviewer comment: 'in a health-related context may prove relevant also for other academic fields such as social or environmental sciences' - I think that much of what you find may be generalisable to other disciplines, so I suggest that you state this earlier as a possibility.

Authors’ response: We agree that our results may indeed prove relevant to other disciplines. However, we decided to narrow our research efforts to the health-related fields, so we think this should be rather a part of the discussion/outlook than earlier in the manuscript. We changed the sentence to underline that aspect: “Although we solely focus on a health-related context, it is possible that some of the recommendations developed during this project may prove relevant also for other academic fields such as social or environmental sciences”.
Competing Interests: None Close
Report a concern

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 01 Dec 2020

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 3 (revision) 02 Sep 21		read
Version 2 (revision) 04 Aug 21	read	read
Version 1 01 Dec 20	read	read

Julie Glanville, Cochrane Information Retrieval Methods Group, London, UK
David Moher, University of Ottawa, Ottawa, Canada; Ottawa Hospital Research Institute, Ottawa, Canada

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

20 Views

23 Sep 2021 | for Version 3

David Moher, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada

20 Views Cite this report Responses(0)

Approved

The authors have addressed the concerns I raised in my second review.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Systematic reviews; open science; reporting guidelines.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

14 Views

18 Aug 2021 | for Version 2

Julie Glanville, Cochrane Information Retrieval Methods Group, London, UK

14 Views Cite this report Responses(0)

Approved

I have read the revised version of the study protocol and I am happy with the changes that have been made.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Information retrieval for evidence identification for systematic reviews.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

26 Views

18 Aug 2021 | for Version 2

David Moher, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada

26 Views Cite this report Responses(1)

Approved With Reservations

I think the revisions, particularly around the scoping review, are much improved.

In terms of the Delphi study, I think details are still missing. For example, it is unclear how many rounds will be given. After round 1, what consensus rate (e.g., 75%) will be used to drop an item from subsequent rounds? Overall, I think more details are required for the proposed Delphi methods. My recommendation is to read these papers:

Examples of Delphi studies: Vogel et al., 2019¹, Santaguida et al., 2018²;
General methodological issues about Delphi: Okoli et al., 2004³, Diamond et al., 2014⁴

One final minor detail. In the study status section of the paper, the authors state they started in November 2020. Is this correct?

References

1. Vogel C, Zwolinsky S, Griffiths C, Hobbs M, et al.: A Delphi study to build consensus on the definition and use of big data in obesity research.Int J Obes (Lond). 43 (12): 2573-2586 PubMed Abstract | Publisher Full Text
2. Santaguida P, Dolovich L, Oliver D, Lamarche L, et al.: Protocol for a Delphi consensus exercise to identify a core set of criteria for selecting health related outcome measures (HROM) to be used in primary health care. BMC Family Practice. 2018; 19 (1). Publisher Full Text
3. Okoli C, Pawlowski S: The Delphi method as a research tool: an example, design considerations and applications. Information & Management. 2004; 42 (1): 15-29 Publisher Full Text
4. Diamond IR, Grant RC, Feldman BM, Pencharz PB, et al.: Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies.J Clin Epidemiol. 2014; 67 (4): 401-9 PubMed Abstract | Publisher Full Text

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Systematic reviews; open science; reporting guidelines.

Respond to this report

Responses (1)

Author Response

02 Sep 2021

Julian Hirt, International Graduate Academy, Institute of Health and Nursing Science, Medical Faculty, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany

Dear David Moher
Thank you very much for your helpful comments and references. We revised our manuscript with respect to methodological details of the Delphi study and now specified the number of rounds we expect to perform, the consensus rate, anonymity of participants, and expected non-response.

Reviewer comment: I think the revisions, particularly around the scoping review, are much improved.
In terms of the Delphi study, I think details are still missing. For example, it is unclear how many rounds will be given. After round 1, what consensus rate (e.g., 75%) will be used to drop an item from subsequent rounds? Overall, I think more details are required for the proposed Delphi methods. My recommendation is to read these papers:
Examples of Delphi studies: Vogel et al., 2019 1, Santaguida et al., 2018 2;
General methodological issues about Delphi: Okoli et al., 2004 3, Diamond et al., 2014 4

Authors’ response: We added “Based on the results of our scoping review, we will formulate draft recommendations for the first Delphi round. Experts will be invited to rate their agreement with the draft recommendations using a four-point Likert scale (strongly agree – agree – disagree – strongly disagree). If experts vote disagree/strongly disagree, they will be required to comment on their reasons and/or give constructive feedback. We consider a recommendation as consented when at least 75% of the experts agree/strongly agree. All other recommendations will be adapted for the next Delphi round. This adaptation will be based on the comments collected from the experts and, if necessary, on discussion via video conference.
There are items where we will not directly propose recommendations, e.g., if the results of our scoping review do not allow it or if there are several equally valid options (e.g., for terminology). In these cases, we will either ask the Delphi experts for their experiences and perspectives or let them vote on several options. We will use the resulting answers to formulate draft recommendations, which will be entered into the Delphi consensus process (see above). Therefore, our Delphi study may comprise qualitative and quantitative aspects.
We will limit the number of Delphi rounds to a maximum of four rounds. Should there be no consensus for any of the items by the end of the fourth round, we will report the results but not give any recommendations.
Expert assessments will be anonymous among experts but open to the study team. We expect a low non-response rate since experts' participation is indicative of their interest in our study.”

One final minor detail. In the study status section of the paper, the authors state they started in November 2020. Is this correct?

Authors’ response: Yes, we started to work on this scoping review in November by doing the initial searches. In the revised version, we reworded to be clearer on that issue: “We conducted the initial search for the scoping review in November 2020 and expect to complete the Delphi study in 2022.”

View more View less

Competing Interests

No competing interests.

Back to all reports

Reviewer Report

28 Views

07 Apr 2021 | for Version 1

David Moher, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada

28 Views Cite this report Responses(1)

Approved With Reservations

A more conceptual question is to what degree this initiative overlaps and differs from PRISMA-S (Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. doi: 10.1186/s13643-020-01542-z)¹.
For example, item 5 of PRIMSA-S – Citation searching - states “Citation searching 5 Indicate whether cited references or citing references were examined, and describe any methods used for locating cited/citing references (e.g., browsing reference lists, using a citation index, setting up email alerts for references citing included studies).” I think this is important so as not to potentially confuse readers. Indeed, should the current initiative be seen as an implementation of PRISMA-S?
I had more difficulty understanding the proposed survey. For me, much of the methods were missing. I understand the recruitment (well reported). I would recommend the authors include the PRISMA-S group unless there is a lot of overlap between PRISMA-S and the other groups mentioned.
It was not clear to me how long the authors will make completing the survey– 15 minutes of 45 minutes? Similarly, will the respondents receive an incentive for completing the survey?
Why ‘simply’ a survey rather than a Delphi (or modified) approach? Will the survey (or Delphi) be pilot tested for question and response option clarity and language?
The data analysis describes cross tables. I assume the authors mean cross tabulations which can be more than descriptive. Will the authors be conducting Chi-square analysis or other analytical approaches of the cross tabulations (e.g., p values)?
The ethical concerns section is likely jurisdiction specific. In my setting ethics would be required. Can the authors explicitly indicate whether ethics is required or not. The section is currently vague on this critical issue.
For the survey, do the authors have an estimated sample size they are aiming for. Similarly, do the authors have an estimated response rate they are aiming for?

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Partly

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Systematic reviews; open science; reporting guidelines.

Respond to this report

Responses (1)

Author Response

04 Aug 2021

Julian Hirt, Institute of Applied Nursing Science, Department of Health, Eastern Switzerland University of Applied Sciences (formerly FHS St.Gallen), St.Gallen, Switzerland

Dear David Moher

We are grateful for your reviewer report and, in particular, for your helpful suggestions to derive evidence-based recommendations from our study. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
You will see that in response to your comments, we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope that these changes fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: A more conceptual question is to what degree this initiative overlaps and differs from PRISMA-S (Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. doi: 10.1186/s13643-020-01542-z).

Authors’ response: PRISMA-S is a reporting guideline for systematic literature searches. However, it is lacking on precise recommendations how to report citation tracking methods and results. By conducting a Delphi study with selected experts in the field (see our answer below for details), we aim at developing more detailed recommendations on adequately reporting citation tracking methods as part of systematic literature searching. This may complement recommendations on reporting given by the PRISMA-S group or could even be integrated in future versions. For the Delphi, we will invite the PRISMA-S group to best use synergies and experiences.

Reviewer comment: For example, item 5 of PRIMSA-S – Citation searching - states “Citation searching 5 Indicate whether cited references or citing references were examined, and describe any methods used for locating cited/citing references (e.g., browsing reference lists, using a citation index, setting up email alerts for references citing included studies).” I think this is important so as not to potentially confuse readers. Indeed, should the current initiative be seen as an implementation of PRISMA-S?

Authors’ response: See answer above. An important detail for the reporting of our study: since we will use citation tracking as part of our information retrieval, we will use PRISMA-S to guide our reporting of the scoping review. In the revised version, we reference it accordingly: “We will document our search strategy according to PRISMA-S”.

Reviewer comment: I had more difficulty understanding the proposed survey. For me, much of the methods were missing. I understand the recruitment (well reported). I would recommend the authors include the PRISMA-S group unless there is a lot of overlap between PRISMA-S and the other groups mentioned.

Authors’ response: As detailed further below, the survey will be replaced by a Delphi study. So, we replaced the survey methods with more detailed methods for the Delphi study. Concerning PRISMA-S, we agree with the reviewer and will ask the PRISMA-S working group to participate in our Delphi study (see below).

Reviewer comment: It was not clear to me how long the authors will make completing the survey– 15 minutes of 45 minutes? Similarly, will the respondents receive an incentive for completing the survey?

Authors’ response: Following your recommendation below, we now plan to conduct an expert Delphi study containing multiple Delphi rounds. We expect that experts will invest around 30 to 90 minutes per Delphi round depending on the underpinning aim of the Delphi round as well as experts’ familiarity and experiences with the topic. Participants will not receive an incentive for participation.

Reviewer comment: Why ‘simply’ a survey rather than a Delphi (or modified) approach? Will the survey (or Delphi) be pilot tested for question and response option clarity and language?

Authors’ response: This is an important point and we thank the reviewer for raising it. We agree that a Delphi study containing several Delphi rounds is suitable to collect the perspectives of international experts on citation tracking, promote discussions on the topic as well as to derive consensus recommendations for future practice and research on the use of citation tracking in systematic literature searching for health-related topics. We subjected our protocol to a major revision based on this point and now outline the methods for the Delphi. As stated in the revised text, we will pilot test and discuss our Delphi items with a person experienced in literature searching but who is not an author and not involved in the Delphi study.

Reviewer comment: The data analysis describes cross tables. I assume the authors mean cross tabulations which can be more than descriptive. Will the authors be conducting Chi-square analysis or other analytical approaches of the cross tabulations (e.g., p values)?

Authors’ response: We revised the corresponding section to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: The ethical concerns section is likely jurisdiction specific. In my setting ethics would be required. Can the authors explicitly indicate whether ethics is required or not. The section is currently vague on this critical issue.

Authors’ response: We added details to the ethical concerns section. With regard to the Swiss Human Research Act, our research does not concern human diseases and the structure and function of the human body. We will therefore not apply for ethical approval of the expert Delphi study.

Reviewer comment: For the survey, do the authors have an estimated sample size they are aiming for. Similarly, do the authors have an estimated response rate they are aiming for?

Authors’ response: This answer concerns the recruitment of experts for the Delphi study, not the survey. We intend to recruit at least 15 participants by using our stepwise approach for recruitment, the person-based approach (contacting authors of pertinent articles identified during the literature search as well as experts from authors’ professional networks) and the organisation-based approach (contacting national and international organisations and systematic review collaborations).

View more View less

Competing Interests

None

Back to all reports

Reviewer Report

50 Views

05 Jan 2021 | for Version 1

Julie Glanville, Cochrane Information Retrieval Methods Group, London, UK

50 Views Cite this report Responses(1)

Approved With Reservations

Introduction: 'These references are usually eligible for inclusion into the review and known at the start of the citation search10–12.' I think this could be reworded for clarity along the following lines:
'These references are usually eligible for inclusion into the review and some may be known at the beginning of the review and others may emerge as eligible records following the main database searches10–12.'
Then perhaps you may not need the sentence that follows the one I am commenting on.
Introduction: 'The taxonomy used to describe the…'. I think it might be better to use a word such as 'terminology' rather than 'taxonomy'.
Figure 1. The figure would be more helpful if it could show the relationships of the indirect citation tracking in the Figure 1b perhaps by colour coding. Please check diagrams in the following references which incorporate chronology into the citation picture as well as showing the indirect citation tracking clearly.
Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731-46.¹
Janssens A, Gwinn M, Brockman JE, Powell K, Goodman M. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol. 2020;20(1):25.²
Introduction: 'However, recent guidance suggests that combining several citation tracking methods (e.g. screening cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing. ' Please cite the guidance.
Introduction: ' It rather depends on a variety of factors. For instance, citation tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)'. These are all important issues and each deserves a bit more description so that readers can understand the differences. Again I think the word 'taxonomy' should be replaced with 'terminology'.
Introduction: current 'topical reviews' - suggest replacing with 'recent reviews on this topic'. Then please add in more citations so that 'reviews' can be supported with more than one reference.
Introduction: 'health-related systematic literature searching'. This occurs three times and for each occurrence I suggest rewording as follows for clarity: 'literature searching for systematic reviews of health and health-related topics'.
Eligibility criteria. 'evidence retrieval method'. Suggest that 'method' is not needed as the sentence has 'means'.
'Eligible studies need to have a health-related context. Studies without an explicitly specified research context are also eligible. ' This seems contradictory. Why are studies in education topics for example not useful if they are looking at the methods rather than the topic? Just including papers where the topic is not clear seems arbitrary when there may be much to learn from a paper even if it has an engineering topic, say. Perhaps reword along the lines of 'although studies undertaken in the context of health-related topics are the main focus, studies of citation tracking in other literatures will also be eligible where the focus is on exploring the methods rather than the subject topic'.
Table 1. Publication Type cannot be 'Any' when there are publication types listed as ineligible. The authors should list eligible study types or use some sort of exception wording.
Information sources. I am not sure 'free web searching via Google Scholar' is a helpful description, perhaps just say 'searches of Google Scholar' here and further down where it is mentioned again.
'For citation tracking, we will use Scopus, as this database seems to cover the largest number of relevant citations for the purpose of our review30. ' There is quite a lot of research that reports that other resources such as Google Scholar or Microsoft Academic provide wider coverage - the authors may wish to consider searching some of the free resources as well (since Scopus alone is not going to find the largest number of results) or finding other reasons for the use of a Scopus only approach.
'MEDBIB-L' - I think there may be a typo here and it is MEDLIB-L? I think it may also need to be corrected in the Dissemination section.
My understanding of the purpose of a protocol is that it should list what will happen and therefore the resources to be searched should be listed and there should be no examples which imply that other things may be added. So there should be no' e.g.'.
'text mining approach' - the chapter describes many approaches - which one was used in this case?
'parts of our textwords' - suggest 'some of our textwords'?
Box 1 - there is harmless redundancy in the search. For example co-citation is searched alone and in combination with other terms - it only needs to be searched alone.
Box 1 - with the focus on searching in the titles, the use of such close adjacency does not seem warranted - it would be more sensitive to use AND for title searching.
Box 1 - it might be helpful in the absence of subject headings and the focus on searching mainly in titles to also search the author keywords field.
Data charting process: I realise it is difficult to know what will happen but the authors description is not very clear about their plans 'We aim for an iterative data extraction process, but in the final publication, we will provide a detailed overview of extracted data items.' What is an 'iterative data extraction process' - what might it look like and what does it involve?
Data collection 'preferred taxonomy' - again, I suggest the word you mean is 'terminology' not 'taxonomy'.
Data analysis. 'To analyse the survey data, we will apply descriptive statistics based on frequencies, percentages, and cross tables.' I think it may be 'cross tabulations' rather than 'cross tables'?
Dissemination of results - it might be useful to also to try to present results at conferences of systematic review experts - Cochrane Colloquium and the Health Technology Assessment International (HTAi) conferences. These organisations also have active information retrieval special interest groups.
Conclusions: 'Depending on the available study landscape' - this seems rather vague to me - it would help the reader if the authors could be more explicit about what they mean.
'consequently, on the quality of clinical care' - I think it might not only be clinical care that could be impacted - it might be any aspect of health care depending on what is being reviewed e.g. a service, a policy, a new method of organising staff etc.
'in a health-related context may prove relevant also for other academic fields such as social or environmental sciences' - I think that much of what you find may be generalisable to other disciplines, so I suggest that you state this earlier as a possibility.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

References

Competing Interests

I conduct training courses in information retrieval for the purpose of systematic reviews and these include courses in citation tracking/citation searches.

Reviewer Expertise

Information retrieval for evidence identification for systematic reviews.

Respond to this report

Responses (1)

Author Response

04 Aug 2021

Julian Hirt, Institute of Applied Nursing Science, Department of Health, Eastern Switzerland University of Applied Sciences (formerly FHS St.Gallen), St.Gallen, Switzerland

Dear Julie Glanville

We are grateful for your reviewer report and, in particular, for your helpful suggestions for improvement of our scoping review protocol. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
Please note that in response to the comments by reviewer 2 (David Moher), we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

We hope our revisions now fully meet your concerns.

Sincerely yours,
Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald

Reviewer comment: Introduction: 'These references are usually eligible for inclusion into the review and known at the start of the citation search10–12.' I think this could be reworded for clarity along the following lines:
'These references are usually eligible for inclusion into the review and some may be known at the beginning of the review and others may emerge as eligible records following the main database searches10–12.'
Then perhaps you may not need the sentence that follows the one I am commenting on.

Authors’ response: We adapted this sentence and the subsequent one according to your suggestion.

Reviewer comment: Introduction: 'The taxonomy used to describe the…'. I think it might be better to use a word such as 'terminology' rather than 'taxonomy'.

Authors’ response: Thank you. We now use terminology instead of taxonomy throughout the manuscript.

Reviewer comment: Figure 1. The figure would be more helpful if it could show the relationships of the indirect citation tracking in the Figure 1b perhaps by colour coding. Please check diagrams in the following references which incorporate chronology into the citation picture as well as showing the indirect citation tracking clearly.
Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731-46.1
Janssens A, Gwinn M, Brockman JE, Powell K, Goodman M. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol. 2020;20(1):25.2

Authors’ response: We have colour coded the 4 citation tracking techniques as suggested. We had already consulted the citation tracking figures of Belter et al. and Janssens. et al. amongst others. Furthermore, we have added text to the figure caption. We feel that our figure is now clearer with respect to chronology.

Reviewer comment: Introduction: 'However, recent guidance suggests that combining several citation tracking methods (e.g. screening cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing.' Please cite the guidance.

Authors’ response: We added a reference.

Reviewer comment: Introduction: ' It rather depends on a variety of factors. For instance, citation tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)'. These are all important issues and each deserves a bit more description so that readers can understand the differences. Again I think the word 'taxonomy' should be replaced with 'terminology'.

Authors’ response: Thank you for this point! We have revised and elaborated this section.

Reviewer comment: Introduction: current 'topical reviews' - suggest replacing with 'recent reviews on this topic'. Then please add in more citations so that 'reviews' can be supported with more than one reference.

Authors’ response: We adapted the sentence according to your suggestion, thank you.

Reviewer comment: Introduction: 'health-related systematic literature searching'. This occurs three times and for each occurrence I suggest rewording as follows for clarity: 'literature searching for systematic reviews of health and health-related topics'.

Authors’ response: Thank you for your proposition. We now use “systematic literature searching for health-related topics“.

Reviewer comment: Eligibility criteria. 'evidence retrieval method'. Suggest that 'method' is not needed as the sentence has 'means'.

Authors’ response: We deleted the word ‘method’.

Reviewer comment: 'Eligible studies need to have a health-related context. Studies without an explicitly specified research context are also eligible.' This seems contradictory. Why are studies in education topics for example not useful if they are looking at the methods rather than the topic? Just including papers where the topic is not clear seems arbitrary when there may be much to learn from a paper even if it has an engineering topic, say. Perhaps reword along the lines of 'although studies undertaken in the context of health-related topics are the main focus, studies of citation tracking in other literatures will also be eligible where the focus is on exploring the methods rather than the subject topic'.

Authors’ response: We agree that it is interesting to learn about citation tracking from other disciplines. However, we have decided to focus on studies in a health-related context because this enables us to (i) consider the context in which the studies were conducted and to (ii) specifically direct our conclusions and recommendations to the health context. Furthermore, we concentrate on health-related studies for practical reasons facing the high number of references that need screening. We deleted the sentence ‘Studies without an explicitly specified research context are also eligible’ to avoid misunderstandings.

On that note we also improved our exclusion criteria by adding this criterion: ‘any study only assessing the benefit of combined search methods (whereas the isolated benefit of citation tracking cannot be extracted)’.

Reviewer comment: Table 1. Publication Type cannot be 'Any' when there are publication types listed as ineligible. The authors should list eligible study types or use some sort of exception wording.

Authors’ response: We now use ‘Any reports of empirical studies’.

Reviewer comment: Information sources. I am not sure 'free web searching via Google Scholar' is a helpful description, perhaps just say 'searches of Google Scholar' here and further down where it is mentioned again.

Authors’ response: We now use ‘web searching’ instead of ‘free web searching’ throughout the manuscript.

Reviewer comment: 'For citation tracking, we will use Scopus, as this database seems to cover the largest number of relevant citations for the purpose of our review30.' There is quite a lot of research that reports that other resources such as Google Scholar or Microsoft Academic provide wider coverage - the authors may wish to consider searching some of the free resources as well (since Scopus alone is not going to find the largest number of results) or finding other reasons for the use of a Scopus only approach.

Authors’ response: We agree and now plan forward citation tracking using a triple approach: Scopus, Web of Science, and Google Scholar. We will iteratively repeat citation tracking on newly identified records to include until no further eligible references will be identified. We now describe this new approach in the methods.

Reviewer comment: 'MEDBIB-L' - I think there may be a typo here and it is MEDLIB-L? I think it may also need to be corrected in the Dissemination section.

Authors’ response: The correct name is MEDIBIB-L (the “I” was missing), a mailing list specific to libraries of medicine in the German-speaking area (library = Bibliothek; https://lists.uni-due.de/mailman/listinfo/medbib-l).

Reviewer comment: My understanding of the purpose of a protocol is that it should list what will happen and therefore the resources to be searched should be listed and there should be no examples which imply that other things may be added. So there should be no' e.g.'.

Authors’ response: We deleted ‘e.g.’

Reviewer comment: 'text mining approach' - the chapter describes many approaches - which one was used in this case?

Authors’ response: We used several tools that were used to improve our search. We now list all the tools that we used during search drafting phase in our manuscript and here:

PubMed PubReMiner: https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi
AntConc 3.5.7 (Windows), developed by Laurence Anthony, Faculty of Science and Engineering, Waseda University, Japan
Yale MeSH Analyzer: http://mesh.med.yale.edu/
TerMine: http://www.nactem.ac.uk/software/termine/
Text Analyzer: https://www.online-utility.org/text/analyzer.jsp
Voyant: https://voyant-tools.org/
VOSviewer: https://www.vosviewer.comrt

Reviewer comment: 'parts of our textwords' - suggest 'some of our textwords'?

Authors’ response: We agree, thank you.

Reviewer comment: Box 1 - there is harmless redundancy in the search. For example co-citation is searched alone and in combination with other terms - it only needs to be searched alone.

Authors’ response: Yes, these are remnants of a previously more specific search and indeed now redundant. Since we already ran the searches by now, we would prefer to leave the search string exactly as we ran it for precise documentation. However, we noted this in our search files in case of a later update.

Reviewer comment: Box 1 - with the focus on searching in the titles, the use of such close adjacency does not seem warranted - it would be more sensitive to use AND for title searching.

Authors’ response: Again, this is very good input which we noted in our search files in case of a later update. Unfortunately, we already ran the searches by now and are done screening. We are, however, confident that any article that we missed due to the close adjacency will be retrieved through our extensive citation tracking, web searches or expert contacting.

Reviewer comment: Box 1 - it might be helpful in the absence of subject headings and the focus on searching mainly in titles to also search the author keywords field.

Authors’ response: Thank you for the sensible suggestion which we also noted in our search files in case of a later update.

Reviewer comment: Data charting process: I realise it is difficult to know what will happen but the authors description is not very clear about their plans 'We aim for an iterative data extraction process, but in the final publication, we will provide a detailed overview of extracted data items.' What is an 'iterative data extraction process' - what might it look like and what does it involve?

Authors’ response: We revised this paragraph and provide more details: ‘Since we expect heterogeneous studies in terms of aim, design, and methods, we aim for an iterative data extraction process. This allows a flexible and study-specific data extraction process, e.g. by adding previously neglected data extraction items that might contribute to the overall body of knowledge to the data extraction form.’

Reviewer comment: Data collection 'preferred taxonomy' - again, I suggest the word you mean is 'terminology' not 'taxonomy'.

Authors’ response: Indeed, thank you.

Reviewer comment: Data analysis. 'To analyse the survey data, we will apply descriptive statistics based on frequencies, percentages, and cross tables.' I think it may be 'cross tabulations' rather than 'cross tables'?

Authors’ response: We revised this paragraph to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: Dissemination of results - it might be useful to also to try to present results at conferences of systematic review experts - Cochrane Colloquium and the Health Technology Assessment International (HTAi) conferences. These organisations also have active information retrieval special interest groups.

Authors’ response: Thank you, we added these conferences to our list of potential conferences.

Reviewer comment: Conclusions: 'Depending on the available study landscape' - this seems rather vague to me - it would help the reader if the authors could be more explicit about what they mean.

Authors’ response: We deleted ‘Depending on the available study landscape’ to be more precise.

Reviewer comment: 'consequently, on the quality of clinical care' - I think it might not only be clinical care that could be impacted - it might be any aspect of health care depending on what is being reviewed e.g. a service, a policy, a new method of organising staff etc.

Authors’ response: We agree and revised the sentence to ‘health care’ instead of ‘clinical care’.

Reviewer comment: 'in a health-related context may prove relevant also for other academic fields such as social or environmental sciences' - I think that much of what you find may be generalisable to other disciplines, so I suggest that you state this earlier as a possibility.

Authors’ response: We agree that our results may indeed prove relevant to other disciplines. However, we decided to narrow our research efforts to the health-related fields, so we think this should be rather a part of the discussion/outlook than earlier in the manuscript. We changed the sentence to underline that aspect: “Although we solely focus on a health-related context, it is possible that some of the recommendations developed during this project may prove relevant also for other academic fields such as social or environmental sciences”.

View more View less

Competing Interests

None

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

Using citation tracking for systematic literature searching - study protocol for a scoping review of methodological studies and an expert survey

Abstract

Keywords

Introduction

Figure 1. Overview of citation tracking methods.

Protocol

Scoping review

Table 1. Inclusion and exclusion criteria.

Box 1. Search strategy for MEDLINE via Ovid

Expert survey

Table 2. Reminder strategy of the online survey.

Dissemination of results

Study status

Conclusions

Data availability

Underlying data

Reporting guidelines

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated