Data extraction methods for systematic review (semi)automation: A living review protocol

Background

Research on systematic review (semi)automation sits at the interface between evidence-based medicine and data science. The capacity of computers for supporting humans increases, along with the development of processing power and storage space. Data extraction for systematic reviewing is a repetitive task. This opens opportunities for support through intelligent software. Tools and methods in this domain frequently focused on automatic processing of information related to the PICO framework (Population, Intervention, Comparator, Outcome). A 2017 analysis of 195 systematic reviews investigated the workload associated with authoring a review. On average, the analysed reviews took 67 weeks to write and publish. Although review size and the number of authors varied between the analysed reviews, the authors concluded that supporting the reviewing process with technological means is important in order to save thousands of personal working hours of trained and specialised staff¹. The potential workload for systematic reviewers is increasing, because the evidence base of clinical studies that can be reviewed is growing rapidly (Figure 1). This entails not only a need to publish new reviews, but also to commit to them and to continually keep the evidence up to date.

Figure 1. Study registrations on ClinicalTrials.gov show an increasing trend.

Challenges in the field of systematic review (semi)automation

Language processing toolkits and machine learning libraries are well documented and available to use free of charge. At the same time, freely available training data make it easy to train classic machine-learning classifiers such as support vector machines, or even complex, deep neural networks such as long short-term memory (LSTM) neural networks. These are reasons why health data science, much like the rest of computer science and natural language processing, is a rapidly developing field. There is a need for fast publication, because trends and state-of-the-art methods are changing at a fast pace. Preprint repositories, such as the arXiv, are offering near rapid publication after a short moderation process rather than full peer review. Consequently, publishing research is becoming easier.

Why this review is needed

An easily updatable review of available methods and tools is needed to inform systematic reviewers, data scientists or their funders alike on the status quo of (semi)automated data extraction methodology. For data scientists, it contributes towards reducing waste and duplication in research. For reviewers, it contributes towards highlighting the current possibilities for data extraction and empowering them to choose the right tools for their tasks in order to work more efficiently. Systematic reviewers are free to use any published tool that is available to them and need sufficient information to make informed decisions about which tools are to be preferred. Therefore, our proposed continuous analysis of the available tools will not only include the final scores that a model achieves, but it will also assess dimensions such as transparency of methods, reproducibility, and how these items are reported. Reported pitfalls of applying health data science methods to systematic reviewing tasks will be summarised to highlight risks that current, as well as future, systems are facing. Reviewing the available literature on systematic review automation is one of many small steps towards supporting evidence synthesis of all available medical research data. If the evidence arising from a study is never reviewed, and as a result never noticed by policy makers and providers of care, then it counts towards waste in research.

Aims of this review

This review aims to:

Related research

We have identified three publications involving reviews of tools and methods, a document providing overviews and guidelines relevant to our topic, and an ongoing effort to characterise published tools for different parts of the systematic reviewing process with respect to interoperability and workflow integration. In 2014, Tsafnat et al.² provided a broad overview on automation technologies for different stages of authoring a systematic review.

A systematic review focusing on text-mining approaches was published in 2015. It includes a summary of methods for the evaluation of systems (such as recall, F1 and related scores). The reviewers focused on tasks related to PICO classification and supporting the screening process³.

A further review of the same year also described methods for data extraction, focusing on PICOs and related fields⁴.

These reviews present an overview of classical machine learning methods applied to tasks such as data mining in the field of evidence-based medicine. At the time of publication of these documents, methods such as topic modelling (Latent Dirichlet Allocation) and support vector machines constituted the state-of-the art for language models. The age of these documents means that the latest static or contextual embedding-based and neural methods are not included. These modern methods, however, are used in contemporary systematic review automation software⁵.

Beller et al.⁶ present a brief overview of tools for systematic review automation. They discuss principles for systematic review automation from a meeting of the International Collaboration for the Automation of Systematic Reviews (ICASR). They highlight that low levels of funding, as well as the complexity of integrating tools for different systematic reviewing tasks have led to many small and isolated pieces of software. A working group formed at the ICASR 2019 Hackathon is compiling an overview of tools published on the Systematic Review Toolbox website⁷. This ongoing work is focused on assessing maintenance status, accessibility and supported reviewing tasks of 120 tools that can be used in any part of the systematic reviewing process as of November 2019.

Prospective registration of this review

We registered this protocol via OSF (https://doi.org/10.17605/OSF.IO/ECB3T). PROSPERO was initially considered as platform for registration, but it is limited to reviews with health related outcomes.

Choosing to maintain this review as a living review

The challenges highlighted in the previous section create several problems. A large variety of approaches and different means of expressing results creates uncertainty in the existing evidence. At the same time, new evidence is likely to emerge. Rapid means of publications necessitate a structured, but at the same time easily updatable review of published methods and tools in the field. We therefore chose a living review approach as the updating strategy for this review.

Search and updates

For literature searches and updates we follow the living review recommendations published by Elliott et al.⁸ and Brooker et al.⁹, as well as F1000Research guidelines for projects that are included in their living evidence collection. We plan to run searches for new studies every second month. This will also include screening abstracts of the newly retrieved reports. The review itself will be updated every six months, providing that a sufficient amount of new records are identified for inclusion. As a threshold for updating, we plan to use 10 new records, but we will consider updating the review earlier if new impactful evidence is published. Figure 2 describes the anticipated reviewing process in more detail.

Figure 2. Continuous updating of the living review.

Our search strategy was developed with the help of an information specialist. Due to the interdisciplinary topic of this review, we plan to search bibliographic databases related to both medicine and computer science. These include Medline via Ovid and Web of Science, as well as the computer science arXiv and the DBLP computer science bibliography. We aim to retrieve publications related to two clusters of search terms. The first cluster includes computational aspects such as data mining, while the second cluster identifies publication related to systematic reviews. The Medline search strategy is provided as Extended data¹⁰. We aim to adapt this search strategy for conducting searches in all mentioned databases. Previous reviews of data mining in systematic reviewing contexts identified the earliest text mining application in 2005^3,4. We therefore plan to search all databases from this year on. In a preliminary test our search strategy was able to identify 4320 Medline records, including all Medline-indexed records included by O’Mara-Eves et al.³. We plan to search the Systematic Review Toolbox website for further information on any published or unpublished tools⁷.

Workflow and study design

All titles and abstracts will be screened independently by two reviewers. Any differences in judgement will be discussed, and resolved with the help of a third reviewer if necessary. The process for assessing full texts will be the same. Data extraction will be carried out by single reviewers, and random 10% samples from each reviewer will be checked independently. If needed, we plan to contact the authors of reports for clarification or further information. In the base review, as well as in every published update, we will present a cross-sectional analysis of the evidence from our searches. This analysis will include the characteristics of each reviewed method or tool, as well as a summary of our findings. Secondly, we will assess the quality of reporting at publication level. This assessment will focus on transparency, reproducibility and both internal and external validity of the described data extraction algorithms. If we at any point deviate from this protocol, we will discuss this in the final publication.

All search results will be de-duplicated and managed with EndNote. The screening and data extraction process will be managed with the help of Abstrackr¹¹ and customised data extraction forms in Excel. All data, including bi-monthly screening results, will be continuously available on our Open ScienceFramework (OSF) repository, as discussed in the Data availability section.

Which systematic reviewing tasks are supported by the methods we review

Tsafnat et al.² categorised sub-tasks in the systematic reviewing process that contained published tools and methods for automation. In our overview, we follow this categorisation and focus on tasks related to data retrieval. More specifically, we will focus on software architectures that receive as input a set of full texts or abstracts of reports. Report types of interest are randomised controlled trials, cohort, or case-control studies. As output, the tools of interest should produce structured data representing features or findings from the study described. A comprehensive list with data fields of interest can be found in the supplementary material for this protocol.

Objectives

Objective 1: to review published methods for data mining and text classification approaches from the data science perspective. This aims at reducing duplicate efforts and encouraging comparability of published methods.

Objective 2: to highlight contributions of data extraction technologies from the perspective of systematic reviewers who wish to use (semi)automation for data extraction. What is the extent of automation, and is it reliable? Can we identify important caveats discussed in the literature?

Eligibility criteria

Included papers

Excluded papers

Outcomes

Primary:

Secondary:

Assessment of the quality of reporting: We will extract information related to the quality of reporting and reproducibility of methods in text mining¹³. The domains of interest, adapted for our reviewing task, are listed in the following.

Dissemination of information

We plan to publish the finished review, along with future updates, via F1000Research.

All data will be available via a project on Open Science Framework (OSF): https://osf.io/4sgfz/ (see Data availability).

Study status

Protocol published. We did a preliminary Medline search as described in this protocol and the supplementary material. The final search, including all additional databases, will be conducted as part of the full review.

Underlying data

No underlying data are associated with this article.

Extended data

Open Science Framework: Data Extraction Methods for Systematic Review (semi)Automation: A Living Review / Protocol. https://doi.org/10.17605/OSF.IO/ECB3T¹⁰

This project contains the following extended data:

Reporting guidelines

Open Science Framework: Data Extraction Methods for Systematic Review (semi)Automation: A Living Review / Protocol. https://doi.org/10.17605/OSF.IO/ECB3T¹⁰

Data are available under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) data waiver.