Using citation tracking for systematic literature searching - study protocol for a scoping review of methodological studies and a Delphi study

Background: Up-to-date guidance on comprehensive study identification for systematic reviews is crucial. According to current recommendations, systematic searching should combine electronic database searching with supplementary search methods. One such supplementary search method is citation tracking. It aims at collecting directly and/or indirectly cited and citing references from "seed references". Tailored and evidence-guided recommendations concerning the use of citation tracking are strongly needed. Objective: We intend to develop recommendations for the use of citation tracking in systematic literature searching for health-related topics. Our study will be guided by the following research questions: What is the benefit of citation tracking for systematic literature searching for health-related topics? Which methods, citation indexes, and other tools are used for citation tracking? What terminology is used for citation tracking methods? Methods: Our study will have two parts: a scoping review and a Delphi study. The scoping review aims at identifying methodological studies on the benefit and use of citation tracking in systematic literature searching for health-related topics with no restrictions on study design, language, and publication date. We will perform database searching in MEDLINE (Ovid), CINAHL (EBSCOhost), Web of Science Core Collection, two information science databases, web searching, and contact experts in the field. Two reviewers will independently perform study selection. We will conduct direct backward and forward citation tracking on included articles. Data from included studies will be extracted using a prespecified extraction sheet and presented in both tabular and narrative form. The results of the scoping review will inform the subsequent Delphi study through which we aim to derive consensus recommendations for the future practice and research of citation tracking.


Introduction
Systematic reviews are considered to be of high clinical and methodological importance as they help to derive recommendations for health care practice and future research [1][2][3] . A comprehensive literature search that aims to identify the available evidence as completely as possible is the foundation of every systematic review 4-6 . Due to an ever-growing research volume, lack of universal terminology and indexation, as well as extensive time requirements for identifying studies in a systematic way, efficient search approaches are required 5,7,8 . According to current recommendations, systematic search approaches should include both electronic database searching and one or several supplementary search methods 9 . Potential supplementary search methods include citation tracking, contacting study authors or experts, handsearching, trial register searching, and web searching 10 . In this study, we focus on citation tracking.
Citation tracking is an umbrella term for multiple methods which directly or indirectly collect related references from so called "seed references". These seed references are usually eligible for inclusion into the review. Some may be known at the beginning of the review and others may emerge as eligible records following full-text screening 10-12 . The terminology used to describe the principles of citation tracking is non-uniform and heterogeneous [13][14][15][16] . Citation tracking methods are subcategorized into direct and indirect citation tracking (Figure 1a). For direct citation tracking, the words "backward" and "forward" denote the directionality of tracking 13,17,18 . Backward citation tracking is the oldest form of citation tracking. It aims at identifying references cited by a seed reference -which can easily be achieved by checking the reference list. Terms like "footnote chasing" or "reference list searching" are synonyms 6,13 . In contrast, forward citation tracking or chaining aims at identifying citing references, i.e. references that cite a seed reference 19 . Indirect citation tracking describes the identification of (i) co-cited references or co-citations (i.e. other references cited by citing literature of a seed reference) and of (ii) co-citing references (i.e. publications sharing references with a seed reference) 11,20 . Direct and indirect citation relationships of references based on a seed reference are illustrated in Figure 1b. Both direct and indirect citation tracking may contain one or more layers of iteration. To this end, researchers may use newly retrieved, relevant references as new seed references.
Direct backward citation tracking of cited references is currently the most common citation tracking method. However, recent guidance suggests that a combination of several methods (e.g., tracking cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing 10 . It is quite likely that the added value of any form of citation tracking is not the same for all systematic reviews. It rather depends on a variety of factors. For instance, citation tracking may be beneficial in research

Amendments from Version 2
We added methodological details of the Delphi study and we now specify the number of rounds we expect to perform, the consensus rate, anonymity of participants, and expected nonresponse.
Any further responses from the reviewers can be found at the end of the article REVISED Figure 1. Overview of citation tracking methods. 1a: Hierarchical illustration of different citation tracking methods; 1b: Direct and indirect citation relationships of references based on a seed reference. A → B denotes A cites B. The horizontal axis denotes time, i.e. the chronology in which references were published relative to the seed reference: "Older" stands for references that were published before the seed reference, "Newer" stands for references that were published after the seed reference.
areas that require complex searches such as reviews of complex interventions, mixed-methods reviews, qualitative evidence syntheses, or reviews on public health topics. Furthermore, research areas without consistent terminology or with vocabulary overlaps with other fields, such as methodological topics, may also benefit from the use of citation tracking 20,21 . Hence, tailored and evidence-guided recommendations on the use of citation tracking are strongly needed. However, none of the current reviews on this topic has systematically identified available evidence on the use and benefit of citation tracking in the context of systematic literature searching 10 .
Therefore, the aim of our study is to develop recommendations for the use of citation tracking in systematic literature searching for health-related topics. The scoping review will be guided by the following three research questions which in turn will inform the Delphi study: • What is the benefit of citation tracking for systematic literature searching for health-related topics?
• Which methods, citation indexes, and other tools are used for citation tracking?
• What terminology is used for citation tracking methods?

Protocol
This protocol is reported according to the "Preferred Reporting Items for Systematic review and Meta-Analysis Protocols" (PRISMA-P) checklist 22 which we published on the Open Science Framework 23 . Our study will have two parts: a scoping review and a Delphi study. The scoping review has the objective to map the benefit and the use of citation tracking, or research gaps if the results are not sufficiently informative. The objective of the subsequent Delphi study is to derive consensus recommendations for future practice and research of citation tracking 24-26 . For the scoping review, we will use the framework by Arksey and O'Malley 26 and the "Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews" (PRISMA-ScR) 27 . For the Delphi study, we will follow the "Guidance on Conducting and REporting DElphi Studies" (CREDES) statement 28 .
Scoping review Eligibility criteria. We will include any study with a focus on citation tracking as a means of evidence retrieval which exhibits one of the following criteria: benefit and/or effectiveness of (i) citation tracking in general; (ii) different methods of citation tracking (e.g., backward vs. forward, direct vs. indirect); or (iii) technical uses of citation tracking (e.g., comparing citation indexes and/or tools, e.g., Scopus vs. Web of Science, Oyster, Voyster). Eligible studies need to have a health-related context. There will be no restrictions on study design, language, and publication date.
We will exclude studies solely using citation tracking for evidence retrieval, e.g., a systematic review applying citation tracking as a supplementary search technique, or studies focussing on citation tracking as a means to explore network or citation impact (i.e. bibliometric analysis). Studies only assessing the benefit of combined search methods in which the isolated benefit of citation tracking cannot be extracted will also be excluded. Furthermore, we will exclude methodological guidelines without empirical investigations and other non-empirical publications like editorials, commentaries, letters and abstract-only publications. Table 1  Additionally, we will perform web searching via Google Scholar as well as direct forward and backward citation tracking of included studies. As some evidence suggests that one citation index may not be enough for this 29 , we will use Scopus, Web of Science, and Google Scholar for forward citation tracking. For backward citation tracking, we will use Scopus and, if seed references are not indexed in Scopus, we will manually extract the seed reference's reference list. We will iteratively repeat direct citation tracking on newly identified eligible references until no further eligible references will be identified. We will also contact librarians in the field of health sciences and information specialists through several mailing lists (Canadian Medical Libraries, Expertsearching, MEDI-BIB-L/German-speaking medical librarians, and EAHIL-list) to ask for further studies.

Search strategy.
Due to a lack of adequate index terms, our search strategy will be based on text words only. To determine frequently occurring terms for inclusion into the search strategy, we analysed keywords in the titles and abstracts of potentially relevant publications retrieved from preliminary searches and similar articles identified via PubMed by using various text mining tools (PubMed Reminer, AntConc, Yale MeSH analyzer, Voyant, VOSviewer, Termine, Text analyzer) 30 . We restricted some of our text words to the title field in order to avoid retrieving systematic reviews that used citation tracking. Data management. A bibliography management tool will be used to manage the number of reference retrievals throughout the study selection process. Furthermore, we will use specific tools for study selection that we describe below.

Selection of sources of evidence.
After an initial calibration phase, that is screening 100 titles and abstracts separately and discussing divergent decisions (TN, JH, HE), two authors (JH, TN) will independently screen titles, abstracts, and full texts using Rayyan 34 . They will solve disagreements by third author arbitration (HE). To screen the results of the citation tracking step, we will consider ASReview, particularly if the number of references exceeds 1000. ASReview combines machine (deep) learning models on a set of eligible studies with active learning on manual selections during title-abstract screening to generate a relevancy-ranked abstract list and to save screening time. Should the tool prove to be beneficial for reducing the screening load, we will consider conducting a more sensitive database search at a later stage and screen additional results with ASReview.
Data charting process. We will pilot a prespecified data extraction sheet approved by consensus among the authors. We will extract bibliographic and geographic data, design-and study-specific data as well as results that answer our research questions. Since we expect heterogeneous studies in terms of aim, design, and methods, we aim for an iterative data extraction process. This will allow a flexible and study-specific data extraction process, e.g., by adding previously neglected data extraction items that might contribute to the overall body of knowledge to the data extraction form. In the final publication, we will provide a detailed overview of extracted data items. One author will extract data and a second author will peer-check the extraction. We will solve disagreements by third author arbitration.

Domain Inclusion criteria Exclusion criteria
Study focus Any study with a focus on citation tracking as an evidence retrieval method AND one of the following criteria: -any study assessing the benefit and/or effectiveness of citation tracking -any study comparing different methods of citation tracking (e.g., backward vs. forward, direct vs. indirect) -any study assessing technical uses of citation tracking (e.g., comparing citation indexes and/or tools, e.g., Scopus vs. WoS, Oyster, Voyster, etc.) Any study solely using citation tracking for evidence retrieval (e.g., a systematic review applying citation tracking as supplementary search technique) OR any study solely assessing benefits and/or use and/or effectiveness of citation tracking to explore a network or citation impact (i.e. bibliometric analysis) OR any study describing solely the method of citation tracking without further assessing it, e.g., guidelines for developing search strategies or guidelines for systematic or other reviews OR any study only assessing the benefit of combined search methods in which the isolated benefit of citation tracking cannot be extracted

Synthesis of results.
One author (JH) will narratively summarise study characteristics and results. Depending on the results, we will also chart them graphically.
Delphi study Design and rationale. A consensus multi-stage online Delphi procedure will be used to derive recommendations for the use of citation tracking in systematic literature searching for health-related topics 28,35,36 . A Delphi procedure will be chosen since the method enables to collect the perspectives of international experts on citation tracking, promote discussions on the topic as well as derive consensus recommendations for future practice and research. The Delphi study will entail several Delphi rounds (see below). The results of the scoping review will inform the initial Delphi round (see below for details).
To distribute the Delphi rounds to the experts, we will use the web-based tool SosciSurvey 37 . The Delphi language will be English.

Expert panel.
The recruitment of experts will be based on a stepwise approach. First, we will contact authors of pertinent articles identified during the literature search as well as experts from our professional networks. This "person-based" approach will help us to identify experts who authored papers, books, comments, and reviews in the field of citation tracking. We will ask the contacted persons to take part in the Delphi study. Second, we will identify and contact relevant national and international organisations as well as systematic review collaborations (e.g., Cochrane groups, Joanna Briggs Institute (JBI), Campbell Collaboration, National Academy of Medicine (NAM), expert information specialists, Evidence Synthesis International, and PRISMA-S working group). This "organisation-based approach" will allow us to reach experts in the field of literature retrieval methods who are potentially using citation tracking without necessarily being the authors of methodological studies (yet). By using this stepwise approach, we intend to recruit at least 15 experts.

Data collection.
In online Delphi rounds, we will seek guidance on various aspects of citation tracking. For example, recommendations on the following aspects could be of particular interest: • Questions on citation tracking that currently cannot be answered and require more research Based on the results of our scoping review, we will formulate draft recommendations for the first Delphi round. Experts will be invited to rate their agreement with the draft recommendations using a four-point Likert scale (strongly agree -agree -disagree -strongly disagree). If experts vote disagree/strongly disagree, they will be required to comment on their reasons and/ or give constructive feedback. We consider a recommendation as consented when at least 75% of the experts agree/strongly agree. All other recommendations will be adapted for the next Delphi round. This adaptation will be based on the comments collected from the experts and, if necessary, on discussion via video conference.
There are items where we will not directly propose recommendations, e.g., if the results of our scoping review do not allow it or if there are several equally valid options (e.g., for terminology). In these cases, we will either ask the Delphi experts for their experiences and perspectives or let them vote on several options. We will use the resulting answers to formulate draft recommendations, which will be entered into the Delphi consensus process (see above). Therefore, our Delphi study may comprise qualitative and quantitative aspects.
We will limit the number of Delphi rounds to a maximum of four rounds. Should there be no consensus for any of the items by the end of the fourth round, we will report the results but not give any recommendations.
Expert assessments will be anonymous among experts but open to the study team. We expect a low non-response rate since experts' participation is indicative of their interest in our study.
To describe experts' characteristics, we will collect sociodemographic data, i.e. professional education and background, current field of work as well as years of experience in literature searching and citation tracking. We expect that experts will invest around 30 to 90 minutes per Delphi round depending on the underpinning aim of the Delphi round as well as experts' familiarity and experiences with the topic. For each Delphi round, we will schedule approximately three weeks for participation. Table 2 illustrates our reminder strategy within a Delphi round. We will pilot test and discuss our Delphi items with a person experienced in literature searching who is not an author and not involved in the Delphi study.
Data analysis. We will use descriptive statistics for votes for which results are numeric or can be converted into numbers.
For free text answers and statements of experts, we will use thematic categorisation 38 .
Ethical concerns. The online Delphi study will contain introductory information on our aims, the Delphi itself, data management and security. We do not expect vulnerability on the part of experts and with regard to the Swiss Human Research Act, our research does not concern human diseases and the structure and function of the human body 39 . We will therefore not apply for ethical approval of the Delphi study. Taking part in the Delphi study will indicate consent to participate. There will be no mandatory participation once an expert consented to participate. Experts will not receive an incentive for participation and may leave the process at any time.

Dissemination of results
Our dissemination strategy uses multiple ways to share our study results with academic stakeholders. The final scoping review and Delphi study will each be published in an international open access journal relevant in the field of information retrieval. Additionally, we will discuss our results with experts at national and international conferences (e.g., conference of the German Network for Evidence-based Medicine (EbM-Netzwerk), conference of the European Association for Health Information and Libraries (EAHIL), Cochrane Colloquium, Health Technology Assessment International (HTAi) conference). To inform about our study results and publications, we will use Twitter, ResearchGate, and mailing lists from relevant stakeholders such as Canadian Medical Libraries, Expertsearching, MEDIBIB-L/German-speaking medical librarians, and EAHIL-list.

Study status
We conducted the initial search for the scoping review in November 2020 and expect to complete the Delphi study in 2022.
Current study status: literature searches: yes; piloting of the study selection process: yes; formal screening of search results against eligibility criteria: yes; data extraction: no; data analysis: no.

Conclusions
Missing pertinent evidence might have an impact on the validity of systematic reviews and, consequently, on the quality of health care 40,41 . Therefore, authors of systematic reviews should conduct high quality literature searches aiming to detect all relevant evidence. Citation tracking may be an effective way to complement electronic database searches and to broaden the scope of possible findings. Therefore, our study intends to provide literature-and expert-based recommendations on the use of citation tracking for systematic literature searching. Although we solely focus on a health-related context, it is possible that some of the recommendations developed during this project may prove relevant also for other academic fields such as social or environmental sciences 9,42 . Finally, tailored and evidence-based recommendations concerning the use of citation tracking for systematic literature searching may guide future steps in semi-automated and automated literature retrieval methods 43,44 .

Underlying data
No underlying data are associated with this article.  In these cases, we will either ask the Delphi experts for their experiences and perspectives or let them vote on several options. We will use the resulting answers to formulate draft recommendations, which will be entered into the Delphi consensus process (see above). Therefore, our Delphi study may comprise qualitative and quantitative aspects. We will limit the number of Delphi rounds to a maximum of four rounds. Should there be no consensus for any of the items by the end of the fourth round, we will report the results but not give any recommendations.

Expert assessments will be anonymous among experts but open to the study team. We expect a low non-response rate since experts' participation is indicative of their interest in our study."
One final minor detail. In the study status section of the paper, the authors state they started in November 2020. Is this correct?
The authors are proposing a new guidance on citation tracking. They will achieve this by completing a scoping review and subsequently a survey. The authors propose using appropriate methods (and reporting) for conducting the scoping review. I found the methods for the survey less clear and hope my comments/questions, below, are helpful in revised the protocol.

1.
For example, item 5 of PRIMSA-S -Citation searching -states "Citation searching 5 Indicate whether cited references or citing references were examined, and describe any methods used for locating cited/citing references (e.g., browsing reference lists, using a citation index, setting up email alerts for references citing included studies)." I think this is important so as not to potentially confuse readers. Indeed, should the current initiative be seen as an implementation of PRISMA-S? 2.
I had more difficulty understanding the proposed survey. For me, much of the methods were missing. I understand the recruitment (well reported). I would recommend the authors include the PRISMA-S group unless there is a lot of overlap between PRISMA-S and 3.
the other groups mentioned.
It was not clear to me how long the authors will make completing the survey-15 minutes of 45 minutes? Similarly, will the respondents receive an incentive for completing the survey?

4.
Why 'simply' a survey rather than a Delphi (or modified) approach? Will the survey (or Delphi) be pilot tested for question and response option clarity and language? 5.
The data analysis describes cross tables. I assume the authors mean cross tabulations which can be more than descriptive. Will the authors be conducting Chi-square analysis or other analytical approaches of the cross tabulations (e.g., p values)? 6.
The ethical concerns section is likely jurisdiction specific. In my setting ethics would be required. Can the authors explicitly indicate whether ethics is required or not. The section is currently vague on this critical issue.

7.
For the survey, do the authors have an estimated sample size they are aiming for. Similarly, do the authors have an estimated response rate they are aiming for?
8. Is the rationale for, and objectives of, the study clearly described? Yes

Are sufficient details of the methods provided to allow replication by others? Partly
Are the datasets clearly presented in a useable and accessible format? Partly We are grateful for your reviewer report and, in particular, for your helpful suggestions to derive evidence-based recommendations from our study. Below, we provide a detailed point-by-point response with changes that we implemented in version 2 of our manuscript.
You will see that in response to your comments, we replaced the planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.

methods were missing. I understand the recruitment (well reported). I would recommend the authors include the PRISMA-S group unless there is a lot of overlap between PRISMA-S and the other groups mentioned.
Authors' response: As detailed further below, the survey will be replaced by a Delphi study. So, we replaced the survey methods with more detailed methods for the Delphi study.
Concerning PRISMA-S, we agree with the reviewer and will ask the PRISMA-S working group to participate in our Delphi study (see below).
Reviewer comment: It was not clear to me how long the authors will make completing the survey-15 minutes of 45 minutes? Similarly, will the respondents receive an incentive for completing the survey?
Authors' response: Following your recommendation below, we now plan to conduct an expert Delphi study containing multiple Delphi rounds. We expect that experts will invest around 30 to 90 minutes per Delphi round depending on the underpinning aim of the Delphi round as well as experts' familiarity and experiences with the topic. Participants will not receive an incentive for participation.

Reviewer comment: Why 'simply' a survey rather than a Delphi (or modified) approach? Will the survey (or Delphi) be pilot tested for question and response option clarity and language?
Authors' response: This is an important point and we thank the reviewer for raising it. We agree that a Delphi study containing several Delphi rounds is suitable to collect the perspectives of international experts on citation tracking, promote discussions on the topic as well as to derive consensus recommendations for future practice and research on the use of citation tracking in systematic literature searching for health-related topics. We subjected our protocol to a major revision based on this point and now outline the methods for the Delphi. As stated in the revised text, we will pilot test and discuss our Delphi items with a person experienced in literature searching but who is not an author and not involved in the Delphi study.
Reviewer comment: The data analysis describes cross tables. I assume the authors mean cross tabulations which can be more than descriptive. Will the authors be conducting Chi-square analysis or other analytical approaches of the cross tabulations (e.g., p values)?
Authors' response: We revised the corresponding section to address a suitable type of analysis for our data retrieved in Delphi rounds. For free text answers and statements of experts, we will use thematic categorisation. For votes whose results are available or can be converted into numbers, we will use descriptive statistics.

Reviewer comment: The ethical concerns section is likely jurisdiction specific. In my setting ethics would be required. Can the authors explicitly indicate whether ethics is required or not. The section is currently vague on this critical issue.
Authors' response: We added details to the ethical concerns section. With regard to the Swiss Human Research Act, our research does not concern human diseases and the structure and function of the human body. We will therefore not apply for ethical approval of the expert Delphi study.
Reviewer comment: For the survey, do the authors have an estimated sample size they are aiming for. Similarly, do the authors have an estimated response rate they are aiming for?

Julie Glanville
Cochrane Information Retrieval Methods Group, London, UK The authors propose to conduct a scoping review of the use of citation tracking techniques to identify research evidence to inform systematic reviews and also to survey experts on their use of citation tracking.
My suggestions for action on the proposal are as follows: Introduction: 'These references are usually eligible for inclusion into the review and known at the start of the citation search10-12.' I think this could be reworded for clarity along the following lines: 'These references are usually eligible for inclusion into the review and some may be known at the beginning of the review and others may emerge as eligible records following the main database searches10-12.' Then perhaps you may not need the sentence that follows the one I am commenting on.

1.
Introduction: 'The taxonomy used to describe the…'. I think it might be better to use a word such as 'terminology' rather than 'taxonomy'.
2. Introduction: 'However, recent guidance suggests that combining several citation tracking methods (e.g. screening cited, citing, co-cited and co-citing references) may be the most effective way to use citation tracking for systematic reviewing. ' Please cite the guidance.

4.
Introduction: ' It rather depends on a variety of factors. For instance, citation tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)'. These are all important issues and each deserves a bit more description so that readers can understand the differences. Again I think the word 'taxonomy' should be replaced with 'terminology'.

5.
Introduction: current 'topical reviews' -suggest replacing with 'recent reviews on this topic'. Then please add in more citations so that 'reviews' can be supported with more than one reference.

6.
Introduction: 'health-related systematic literature searching'. This occurs three times and for each occurrence I suggest rewording as follows for clarity: 'literature searching for systematic reviews of health and health-related topics'.

7.
Eligibility criteria. 'evidence retrieval method'. Suggest that 'method' is not needed as the sentence has 'means'.

8.
'Eligible studies need to have a health-related context. Studies without an explicitly specified research context are also eligible. ' This seems contradictory. Why are studies in education topics for example not useful if they are looking at the methods rather than the topic? Just including papers where the topic is not clear seems arbitrary when there may be much to learn from a paper even if it has an engineering topic, say. Perhaps reword along the lines of 'although studies undertaken in the context of health-related topics are the main focus, studies of citation tracking in other literatures will also be eligible where the focus is on exploring the methods rather than the subject topic'. 'For citation tracking, we will use Scopus, as this database seems to cover the largest number of relevant citations for the purpose of our review30. ' There is quite a lot of 12.
research that reports that other resources such as Google Scholar or Microsoft Academic provide wider coverage -the authors may wish to consider searching some of the free resources as well (since Scopus alone is not going to find the largest number of results) or finding other reasons for the use of a Scopus only approach.
'MEDBIB-L' -I think there may be a typo here and it is MEDLIB-L? I think it may also need to be corrected in the Dissemination section.

13.
My understanding of the purpose of a protocol is that it should list what will happen and therefore the resources to be searched should be listed and there should be no examples which imply that other things may be added. So there should be no' e.g.'. 14.
'text mining approach' -the chapter describes many approaches -which one was used in this case? 15.
Box 1 -there is harmless redundancy in the search. For example co-citation is searched alone and in combination with other terms -it only needs to be searched alone.

17.
Box 1 -with the focus on searching in the titles, the use of such close adjacency does not seem warranted -it would be more sensitive to use AND for title searching.

18.
Box 1 -it might be helpful in the absence of subject headings and the focus on searching mainly in titles to also search the author keywords field.

19.
Data charting process: I realise it is difficult to know what will happen but the authors description is not very clear about their plans 'We aim for an iterative data extraction process, but in the final publication, we will provide a detailed overview of extracted data items.' What is an 'iterative data extraction process' -what might it look like and what does it involve? 20.
Data collection 'preferred taxonomy' -again, I suggest the word you mean is 'terminology' not 'taxonomy'.

21.
Data analysis. 'To analyse the survey data, we will apply descriptive statistics based on frequencies, percentages, and cross tables.' I think it may be 'cross tabulations' rather than 'cross tables'?

22.
Dissemination of results -it might be useful to also to try to present results at conferences of systematic review experts -Cochrane Colloquium and the Health Technology Assessment International (HTAi) conferences. These organisations also have active information retrieval special interest groups.

23.
Conclusions: 'Depending on the available study landscape' -this seems rather vague to meit would help the reader if the authors could be more explicit about what they mean.

24.
'consequently, on the quality of clinical care' -I think it might not only be clinical care that 25. could be impacted -it might be any aspect of health care depending on what is being reviewed e.g. a service, a policy, a new method of organising staff etc.
'in a health-related context may prove relevant also for other academic fields such as social or environmental sciences' -I think that much of what you find may be generalisable to other disciplines, so I suggest that you state this earlier as a possibility.

26.
planned survey with a planned Delphi study to address the original objective. This change also entails the reformulation of our research questions and the intent to publish the final scoping review and the results of the Delphi study separately.
We hope our revisions now fully meet your concerns.
Sincerely yours, Julian Hirt, Thomas Nordhausen, Christian Appenzeller-Herzog, and Hannah Ewald Reviewer comment: Introduction: 'These references are usually eligible for inclusion into the review and known at the start of the citation search10-12.' I think this could be reworded for clarity along the following lines: 'These references are usually eligible for inclusion into the review and some may be known at the beginning of the review and others may emerge as eligible records following the main database searches10-12.' Then perhaps you may not need the sentence that follows the one I am commenting on.
tracking may be especially beneficial in case of (i) complex searches (e.g. for reviews on public health topics), (ii) searching for health outcome measurement instruments, or (iii) research areas without consistent taxonomy, with vocabulary overlaps with other fields, or with a lack of index terms in databases (e.g. methodological topics)'. These are all important issues and each deserves a bit more description so that readers can understand the differences. Again I think the word 'taxonomy' should be replaced with 'terminology'. Authors' response: We agree, thank you.
Reviewer comment: Box 1 -there is harmless redundancy in the search. For example co-citation is searched alone and in combination with other terms -it only needs to be searched alone.