This review is, to the best of our knowledge, the only living systematic review (LSR) of data extraction methods in clinical trial text. A living review of automated data extraction for social science studies was published recently, adapting part of our methodology. ¹⁵⁵ We identified four previous reviews of tools and methods in the first iteration of this living review (called base-review hereafter), ^{2 – 5} and two documents providing overviews and guidelines relevant to our topic. ^{3 , 6 , 7} Between the base-review and the 2023 update, we identified six more related (systematic) literature reviews. ^{8 – 13} For the most recent 2024 update, 13 reviews and seven editorials or opinion pieces were identified.

Related reviews before 2018: The systematic reviews from 2014 to 2015 present an overview of classical machine learning and natural language processing (NLP) methods applied to tasks such as data mining in the field of evidence-based medicine. At the time of publication of these documents, methods such as topic modelling (Latent Dirichlet Allocation) and support vector machines (SVM) were considered state-of-the art for language models.

In 2014, Tsafnat et al. provided a broad overview on automation technologies for different stages of authoring a systematic review. ⁵ O’Mara-Eves et al. published a systematic review focusing on text-mining approaches in 2015. ⁴ It includes a summary of methods for the evaluation of systems, such as recall, accuracy, and F1 score (the harmonic mean of recall and precision, a metric frequently used in machine-learning). The reviewers focused on tasks related to PICO classification and supporting the screening process. In the same year, Jonnalagadda, Goyal and Huffman ³ described methods for data extraction, focusing on PICOs and related fields. The age of these publications means that the latest static or contextual embedding-based and neural methods are not included. These newer methods, ¹⁴ however, are used in contemporary systematic review automation software which will be reviewed in the scope of this living review.

Related reviews up to 2020: Reviews up to 2020 focus on discussions around tool development and integration in practice, and mark the starting date of the inclusion of automation methods based on neural networks. Beller et al. describe principles for development and integration of tools for systematic review automation. ⁶ Marshall and Wallace ⁷ present a guide to automation technology, with a focus on availability of tools and adoption into practice. They conclude that tools facilitating screening are widely accessible and usable, while data extraction tools are still at piloting stages or require a higher amount of human input.

A systematic review of machine-learning for systematic review automation, published in Portuguese in 2020, included 35 publications. The authors examined journals in which publications about systematic review automation are published, and conducted a term-frequency and citation analysis. They categorised papers by systematic review task, and provided a brief overview of data extraction methods. ²

Related reviews up to 2023 update: These six reviews include and discuss end-user tools and cover different tasks across the SR workflow, including data extraction. Compared with this LSR, these reviews are broader in scope but have less included references on the automation of data extraction. Ruiz and Duffy ¹⁰ did a literature and trend analysis showing that the number of published references about SR automation is steadily increasing. Sundaram and Berleant ¹¹ analyse 29 references applying text mining to different parts of the SR process and note that 24 references describe automation in study selection while research gaps are most prominent for data extraction, monitoring, quality assessment, and synthesis. ¹¹ Khalil et al. ⁹ include 47 tools and descriptions of validation studies in a scoping review, of which 8 are available end-user tools that mostly focus on screening, but also cover data extraction and risk of bias assessments. They discuss limitations of tools such as lack of generalisability, integration, funding, and limited performance or access. ⁹ Cierco Jimenez et al. ⁸ included 63 references in a mapping review of machine-learning to assist SRs during different workflow steps, of which 41 were available end-user tools for use by researchers without informatics background. In accordance with other reviews they describe screening as the most frequently automated step, while automated data extraction tools are lacking due to the complexity of the task. Zhang et al. ¹² included 49 references on automation of data extraction fields such as diseases, outcomes, or metadata. They focussed on extraction from traditional Chinese medicine texts such as published clinical trial texts, health records, or ancient literature. ¹² Schmidt et al. ¹³ published a narrative review of tools with a focus on living systematic review automation. They discuss tools that automate or support the constant literature retrieval that is the hallmark of LSRs, while well-integrated (semi) automation of data extraction and automatic dissemination or visualisation of results between official review updates is supported by some, but less common.

Related reviews since 2023 update: We identified a further 13 reviews on the topic of literature review automation, and seven opinion pieces or editorials. All references are listed in Appendix 3.1. We mention here only selected papers due to the large increase in related literature. Aletaha et al. (2023) ¹²⁹ published a highly related scoping review of automated data extraction methods, including 26 references up to 2022. Their conclusions reproduce the conclusions of our previous 2023 LSR update, namely low availability of software and trends towards Transformer models. ¹²⁹

Tóth et al. ¹⁷⁰ and Ofori-Boateng et al. ¹⁶⁰ discussed data extraction among automation methods for other SR tasks. Tóth included 13 data extraction methods and described 15 automated SRs, of these SRs only one automated data extraction while the others employed search and screening methods. ¹⁷⁰ Ofori-Boateng included 52 papers, with six addressing data extraction. ¹⁶⁰

Hammer et al. ¹⁴² reviewed deduplication tools and evaluation methods. In the field of large language models (LLMs) Wang et al. ¹⁷² reviewed prompt engineering methods in the medical field, for example in data extraction or evidence inference. Tam et al. ¹⁶⁸ reviewed 142 papers on the human evaluation of LLM application in healthcare in general and find that generalizability, applicability, and reliability are lacking in current evaluation practices; a finding supported by our current review update.

We aim to review published methods and tools aimed at automating or (semi) automating the process of data extraction in the context of a systematic review of medical research studies. We do this in the form of a living systematic review, keeping information up to date and relevant to the challenges faced by systematic reviewers at any time.

Our objectives in reviewing this literature are two-fold. First, we want to examine the methods and tools from the data science perspective, seeking to reduce duplicate efforts, summarise current knowledge, and encourage comparability of published methods. Second, we seek to highlight the added value of the methods and tools from the perspective of systematic reviewers who wish to use (semi) automation for data extraction, i.e., what is the extent of automation? Is it reliable? We address these issues by summarising important caveats discussed in the literature, as well as factors that facilitate the adoption of tools in practice.

This review was conducted following a preregistered and published protocol. ¹⁵ Any deviations from the protocol have been described below.

We are conducting a living review because the field of systematic review (semi) automation is evolving rapidly along with advances in language processing, machine-learning and deep-learning.

The process of updating started as described in the protocol ¹⁵ and was adapted between review updates. The living review application used for daily reference updates in the past is no longer used, in part due to its programming and packages ageing and becoming unreliable. For example, we observed discrepancies between results retrieved by our automated PubMed search vs. results when applying a search manually via PubMed. ArXiv, ACL, and dblp search updates are still executed and deduplicated using the previous methods, but are then fed manually into SWIFT-ActiveScreener for screening once a year, in order to use priority screening with early stopping as described elsewhere in detail. ¹⁴⁴

The decision for full review updates is made every six months based on the number of new publications added to the review. For more details about this, please refer to the protocol or to the Cochrane living systematic review guidance. Between updates, the screening process and current state of the data extraction is visible via the living review website.

•

We included full text publications that describe an original NLP approach for extracting data related to systematic reviewing tasks. Data fields of interest (referred to here as entities or as sentences) were adapted from the Cochrane Handbook for Systematic Reviews of Interventions, ¹ and are defined in the protocol. ¹⁵ We included the full range of NLP methods (e.g., regular expressions, rule-based systems, machine learning, and deep neural networks).

We excluded publications reporting: •

Methods and tools related solely to image processing and importing biomedical data from PDF files without any NLP approach, including data extraction from graphs.

Base-review: We searched five electronic databases, using the search methods previously described in our protocol. ¹⁵ In short, we searched MEDLINE via Ovid, using a search strategy developed with the help of an information specialist, and searched Web of Science Core Collection and IEEE using adaptations of this strategy, which were made by the review authors. Searches on the arXiv (computer science) and dblp were conducted on full database dumps using the search functionality described by McGuinness and Schmidt. ¹⁶ The full search results and further information about document retrieval are available in Underlying data: Appendix A and B. ¹²⁷

Originally, we planned to include a full literature search from the Web of Science Core Collection. Due to the large number of publications retrieved via this search (n = 7822) we decided to screen publications from all other sources first, to train a machine-learning ensemble classifier, and to add only publications that were predicted as relevant for our living review. This reduced the Web of Science Core Collection publications to 547 abstracts, which were added to the studies in the initial screening step. The dataset, code and weights of trained models are available in Underlying data: Appendix C. ¹²⁷ This includes plots of each model’s evaluation in terms of area under the curve (AUC), accuracy, F1, recall, and variance of cross-validation results for every metric.

Update 1 (2023): As planned, we changed to the PubMed API for searching MEDLINE. This decision was made to facilitate continuous reference retrieval. We searched only for pre-print or published literature and therefore did not search sources such as GITHUB or other source code repositories. We also searched arXiv (computer science), ACL-Anthology, dblp, and used EPPI-Reviewer to collect citations from MicrosoftAcademic which later became OpenAlex. In EPPI-Reviewer we used the ‘Bi-Citation AND Recommendations’ method.

Update 2 (2024): We noticed discrepancies between the automated reference retrieval in PubMed and decided to 1) adjust the search strategy to include LLM-related terms and 2) to re-run the search from July 2021 to retrieve articles potentially missed. For the EPPI-Reviewer/OpenAlex search, we used the same retrieval method but applied it only once in September 2024, after supplying it with new included references from this current review update so that it could retrieve the latest related publications.

2.5.1 Selection of studies

Base review: Initial screening and data extraction were conducted as stated in the protocol. In short, for the base-review we screened all retrieved publications using the Abstrackr tool. All abstracts were screened by two independent reviewers. Conflicting judgements were resolved by the authors who made the initial screening decisions. Full texts screening was conducted in a similar manner to abstract screening but used our web application for LSRs described in the following section.

Update 1 (2023): For the updated review we used our living review web application to retrieve all publications with the exception of the items retrieved by EPPI-Reviewer (these are added to the dataset separately). We further used our application to de-duplicate, screen, and data-extract all publications.

A methodological update to the screening process included a change to single-screening to assess eligibility on both abstract and full-text level, reducing dual-screening to 10% of the publications.

Update 2 (2024): All references from database searches were imported to SWIFT-ActiveScreener and screened to an estimated recall of 95%. ¹⁴⁴ We used included references from the previous LSR update as seeds for the tool’s reference prioritisation algorithm. References retrieved by EPPI-Reviewer from OpenAlex were later screened in full.

2.5.2 Data extraction, assessment, and management

Base Review and update 1 (2023): We previously developed a web application to automate reference retrieval for living review updates (see Software availability ¹⁷ ), to support both abstract and full text screening for review updates, and to manage the data extraction process throughout. ¹⁷ For future updates of this living review we will use the web application, and not Abstrackr, for screening references. This web application is already in use by another living review. ¹⁸ It automates daily reference retrieval from the included sources and has a screening and data extraction interface. All extracted data are stored in a database. Figures and tables can be exported on a daily basis and the progress in between review updates is shared on our living review website. The full spreadsheet of items extracted from each included reference is available in the Underlying data. ¹²⁷ As previously described in the protocol, quality of reporting and reproducibility was initially assessed based on a previously published checklist for reproducibility in text mining, but some of the items were removed from the scope of this review update. ¹⁹

Update 2 (2024): All data extraction was carried out in SWIFT-ActiveScreener.

As planned in the protocol, a single reviewer conducted data extraction, and a random 10% of the included publications were checked by a second reviewer.

2.5.3 Visualisation

Base Review and update 1 (2023): The creation of all figures and interactive plots on the living review website and in this review’s ‘Results’ section was automated based on structured content from our living review database (see Appendix A, D, E Underlying data ¹²⁷ ). We automated the export of PDF reports for each included publication. Calculation of percentages, export of extracted text, and creation of figures was also automated.

Update 2 (2024): We merged data extracted via the different tools with our previous database in order to use the same workflow for visualisation. We additionally created an EPPI-Mapper map [ 1] to display results (available on the website and in Appendix 3.2).

2.5.4 Accessibility of data

All data and code are free to access. A detailed list of sources is given in the ‘Data availability’ and ‘Software availability’ sections.

In the protocol we stated that data would be available via an OSF repository. Instead, the full review data are available via the Harvard Dataverse, as this repository allows us to keep an assigned DOI after updating the repository with new content for each iteration of this living review. We also stated that we would screen all publications from the Web of Science search. Instead, we describe a changed approach in the Methods section, under ‘Search’. For review updates, Web of Science was dropped and replaced with OpenAlex searches via EPPI-Reviewer.

For update 2 we added data extraction items specific to LLM automation, including prompt development, reproducibility, strategies of applying LLMs, and a question about whether the paper describes a study within a review. ¹³³

We added a data extraction item for the type of information which a publication mines (e.g. P, IC, O) into the section of primary items of interest, and we moved the type of input and output format from primary to secondary items of interest. We grouped the secondary item of interest ‘Other reported metrics, such as impacts on systematic review processes (e.g., time saved during data extraction)’ with the primary item of interest ‘Reported performance metrics used for evaluation’.

The item ‘Persistence: is the dataset likely to be available for future use?’ was changed to: ‘Can data be retrieved based on the information given in the publication?’. We decided not to speculate if a dataset is likely to be available in the future and chose instead to record if the dataset was available at the time when we tried to access it.

The item ‘Can we obtain a runnable version of the software based on the information in the publication?’ was changed to ‘Is an app available that does the data mining, e.g. a web-app or desktop version?’.

In the base-review we assessed the included publications based on a list of 17 items in the domains of reproducibility (3.4.1), transparency (3.4.2), description of testing (3.4.3), data availability (3.4.4), and internal and external validity (3.4.5). The list of items was reduced to six items: •

3.4.2.2 Is there a description of the dataset used and of its characteristics?

The following items were removed, although the results and discussion from the assessment of these items in the base-review remains within the review text: •

3.4.1.1 Are the sources for training/testing data reported?

Our database searches identified 10,107 publications after duplicates were removed (see Figure 2). We identified one more publication manually.

This iteration of the living review includes 117 publications, summarised in Table A1 in Underlying data ¹²⁷ ).

3.1.1 Excluded publications

Across the base-review and the updates, 255 publications were excluded at the full text screening stage, with the most common reason for exclusion being that it did not fit target entities or target data. In most cases, this was due to the text-types mined in the publications. Electronic health records and non-trial data were common, and we created a list of datasets that would be excluded in this category (see more information in Underlying data: Appendix B ¹²⁷ ). Some publications addressed the right kind of text but were excluded for not mining data of interest to this review. For example, Norman, Leeflang and Névéol ²³ performed data extraction for diagnostic test accuracy reviews, but focused on extracting the results and data for statistical analyses. Millard, Flach and Higgins ²⁴ and Marshall, Kuiper and Wallace ²⁵ looked at risk of bias classification, which is beyond the scope of this review. Boudin, Nie and Dawes ²⁶ developed a weighing scheme based on an analysis of PICO element locations, leaving the detection of single PICO elements for future work. Luo et al. ²⁷ extracted data from clinical trial registrations but focused on parsing inclusion criteria into event or temporal entities to aid participant selection for randomised controlled trials (RCTs).

The second most common reason for study exclusion was that they had ‘no original data extraction approach’. Rathbone et al., ²⁸ for example, used hand-crafted Boolean searches specific to a systematic review’s PICO criteria to support the screening process of a review within Endnote. We classified this article as not having any original data extraction approach because it does not create any structured outputs specific to P, IC, or O. Malheiros et al. ²⁹ performed visual text mining, supporting systematic review authors by document clustering and text highlighting. Similarly, Fabbri et al. ³⁰ implemented a tool that supports the whole systematic review workflow, from protocol to data extraction, performing clustering and identification of similar publications. Other systematic reviewing tasks that can benefit from automation but were excluded from this review are listed in Underlying data: Appendix B. ¹²⁷

3.2.1 Automation approaches used

Figure 3 shows aspects of the system architectures implemented in the included publications. A short summary of these for each publication is provided in Table A1 in Underlying data. ¹²⁷ Where possible, we tried to break down larger system architectures into smaller components. For example, an architecture combining a word embedding + long short-term memory (LSTM) network would have been broken down into the two respective sub-components. We grouped binary classifiers, such as naïve Bayes and logistic regression. Although SVM is also binary classifier, it was assigned as separate category due to its popularity. The final categories are a mixture of non-machine-leaning automation (application programming interface (API) and metadata retrieval, PDF extraction, rule-base), classic machine-learning (naïve Bayes, decision trees, SVM, or other binary classifiers) and neural or deep-learning approaches (convolutional neural network (CNN), LSTM, transformers, or word embeddings). This figure shows that there is no obvious choice of system architecture for this task. For the LSR update, the strongest trend was the increasing application of LLMs, which appeared in 17 publications. LLMs are large language models such as GPT-4 that were initially intended to generate text, but are also being applied to data extraction tasks. Further LLM training and fine-tuning methods such as LoRA (Low Rank Adaptation) among others were reported in six publications, but fine-tuning was used less frequently than zero-shot prompting with 13 papers or k-shot prompting in two. ^{138 , 148} Previously, BERT (Bidirectional Encoder Representations from Transformers) was the most commonly used architecture, sometimes coupled with CRF or LSTM. BERT was published in 2018 and other architecturally-identical versions of it tailored to using scientific text, such as SciBERT, are summarised under the same category in this review. ^{14 , 31} In the previous update it appeared 21 times while now it is used in 40 included publications. Other transformer-based architectures such as the bio-pretrained version of ELECTRA, are also still gaining attention, ^{32 , 33} as well as FLAIR-based models. ^{34 – 36}

Rule-bases, including approaches using heuristics, wordlists, and regular expressions, were one of the earliest techniques used for data extraction in EBM literature. Rule-bases are still being used, but most publications use them in combination with other classifiers (data shown in Underlying data ¹²⁷ ). Although used more frequently in the past, the 15 publications published between 2017 and now that use this approach alongside other architectures such as LLM, ¹⁴⁸ Transformer, ^{33 , 37 – 39 , 148 , 156 , 174} conditional random fields (CRF), ^{40 , 156} use it with SVM ⁴¹ or other binary classifiers. ⁴² In practice, these systems use rule-bases in the form of hand-crafted lists to identify candidate phrases for amount entities such as sample size ^{42 , 43} or to refine a result obtained by a machine-learning classifier on the entity level (e.g., instances where a specific intervention or outcome is extracted from a sentence). ⁴⁰

Binary classifiers, most notably naïve Bayes and SVMs, are also frequently used system components in the data extraction literature. They are frequently used in studies published between 2005 and now but their usage started declining with the advent of neural models.

Embedding and neural architectures are increasingly being used in literature over the past seven years. Recurrent neural networks (RNN), CNN, and LSTM networks require larger amounts of training data; by using transformer-based embeddings with pre-training algorithms based on unlabelled data they have become increasingly more interesting in fields such as data extraction for EBM- where high-quality training data are difficult and expensive to obtain.

In the ‘Other’ category, tools mentioned were mostly other classifiers such as maximum entropy classifiers (n = 3), kLog, J48, and various position or document-length classification algorithms. We also added inovel training approaches to existing neural architectures in this category, as well as ensemble or normalisation models and custom algorithms like a template-filling algorithm. ^{175 , 176}

3.2.2 Reported performance metrics used for evaluation

Precision (i.e., positive predictive value), recall (i.e., sensitivity), and F1 score (harmonic mean of precision and recall) are the most widely used metrics for evaluating classifiers. This is reflected in Figure 4, which shows that at least one of these metrics was used in the majority of the included publications. Accuracy and area under the curve - receiver operator characteristics (AUC-ROC) were less frequently used.

There were several approaches and justifications of using macro- or micro-averaged precision, recall, or F1 scores in the included publications. Micro or macro scores are computed in multi-class cases, and the final scores can differ whenever the classes in a dataset are imbalanced (as is the case in most datasets used for automating data extraction in SR automation).

Both micro and macro scores were reported by Singh et al. (2021), ⁴⁵ Kilicoglu et al. (2021), ³⁸ Kiritchenko et al. (2010), ⁴⁶ Fiszman et al. (2007), ⁴⁷ Zhang et al. (2024), ¹⁷⁹ Karystianis et al. (2014, 2017) ^{48 , 49} reported micro across documents, and macro across the classes. Jiang et al. (2024) ¹⁴⁸ provide an interesting discussion on the influence of class imbalance on micro vs. macro-scoring and how both approaches can be used to evaluate different aspects of their work.

Macro-scores were previously used in only one publication, ³⁷ but in the current review update seven more publications used them exclusively. ^{130 , 131 , 132 , 139 , 169 , 180 , 181}

Micro scores were used by Fiszman et al. ⁴⁷ for class-level results. In one publication harmonic mean was used for precision and recall, while micro-scoring was used for F1. ⁵⁰ Micro scores were most widely used, including Al-Hussaini et al. (2022), ³² Sanchez-Graillet et al. (2022), ⁵¹ Kim et al. (2011), ⁵² Verbeke et al. (2012), ⁵³ and Jin and Szolovits (2020) ⁵⁴ were used in the evaluation script of Nye et al. (2018). ⁵⁵ In the review update, five more publications applied micro scores. ^{140 , 145 , 151 , 175 , 176}

In the latest update, four publications used weighed or average scores instead. ^{147 , 162 , 167 , 178}

In the category ‘Other’ we added several instances where a relaxation of a metric was introduced, e.g., precision using top-n classified sentences ^{44 , 46 , 56} or mean average precision and the metric ‘precision @rank 10’ for sentence ranking exercises. ^{57 , 58} Another type of relaxation for standard metrics is a distance relaxation when normalising entities into concepts in medical subject headings (MesH) or unified medical language system (UMLS), to allow N hops between predicted and target concepts. ⁵⁹

The LSR update showed an increasing trend of text summarisation and relation extraction algorithms. ROGUE, ∆EI, or Jaccard similarity were metrics for summarisation. ^{60 , 61} For relation extraction F1, precision, and recall remained the most common metrics. ^{62 , 63}

Other metrics were kappa, ⁵⁸ random shuffling ⁶⁴ or binomial proportion test ⁶⁵ to test statistical significance, given with confidence intervals. ⁴¹ Further metrics included under ‘Other’ were odds ratios, ⁶⁶ normalised discounted cumulative gain, ^{44 , 67} ‘sentences needed to screen per article’ in order to find one relevant sentence, ⁶⁸ McNemar test, ⁶⁵ C-statistic (with 95% CI) and Brier score (with 95% CI). ⁶⁹ Barnett (2022) ⁷⁰ extracted sample sizes and reported the mean difference between true and extracted numbers.

Real-life evaluations, such as the percentage of outputs needing human correction, or time saved per article, were reported by four publications, ^{32 , 46 , 150 , 158} and an evaluation as part of a wider screening system was done in another. ⁷¹ Notably, one of these papers evaluated their method in terms of helpfulness and time-taken and did a direct comparison with the existing Trialstreamer application, giving useful insights into practical aspects of using automation tools. ¹⁵⁰

3.2.3 Type of data

3.2.3.1 Scope and data

Most data extraction is carried out on abstracts (See Table A1 in Underlying data, ¹²⁷ and the supplementary table giving an overview of all included publications). Abstracts are the most practical choice, due to the possibility of exporting them along with literature search results from databases such as MEDLINE. Within the 30 (26%) references that reported usage of full texts, most specifically mentioned that this also included abstracts. Due to unclear descriptions and lack of dataset publication it is unclear if all full texts included abstract text, but we assumed that all full texts included abstracts, and that all datasets including abstracts also included titles. Descriptions of the benefits of using full texts for data extraction include having access to a more complete dataset, while the benefits of using titles (N=4, 5%) include lower complexity for the data extraction task. ⁴³ Xu et al. (2010) ⁷² exclusively used titles, while the three publications that specifically mentioned titles also used abstracts in their datasets. ^{43 , 73 , 74}

Figure 5 shows that RCTs are the most common study design texts used for data extraction in the included publications (see also extended Table A1 in Underlying data). ¹²⁷ This is not surprising, because systematic reviews of interventions are the most common type of systematic review, and they are usually focusing on evidence from RCTs. Therefore, the literature for automation of data extraction focuses on RCTs, and their related PICO elements. Systematic reviews of diagnostic test accuracy are less frequent and currently, 5 (4%) publications report using data from diagnostic test papers. Previously only one included publication specifically focused on text and entities related to these studies, ⁷⁵ while two mentioned diagnostic procedures among other fields of interest. ^{35 , 76} During the 2024 update, two more publications were identified that included diagnostic test studies in their corpus among other study types, but only ¹⁶² specifically mined entities related to diagnostic tests. ^{151 , 162} Twelve publications focused on extracting data specifically from epidemiology research, non-randomised interventional studies, or included text from cohort studies as well as RCT text. ^{48 , 49 , 61 , 72 – 74 , 76 , 77 , 151 , 162 , 165 , 166} More publications mining data from surveys, animal RCTs, or case series might have been found if our search and review had concentrated on these types of texts.

3.2.3.2 Data extraction targets

Due to the high numbers of references cited in this section, we removed references for entities that appeared more than 10 times. The publications are still accessible and can be filtered via the map available on the review website ( https://l-ena.github.io/living_review_data_extraction/) and in appendix 3.2. Mining P, IC, and O elements is the most common task performed in the literature of systematic review (semi-)automation (see Table A1 in Underlying data, ¹²⁷ and Figure 6). In the base-review, P was the most common entity. Currently, O (n=85, 72%) has become the most popular, due to the continuing trend of relation-extraction models that focus on the relationship between O and I entities and therefore may omit the automatic extraction of P. Some of the less-frequent data extraction targets in the literature can be categorised as sub-classes of a PICO, ⁵⁵ for example, by annotating hierarchically multiple entity types such as health condition, age, and gender under the P class. The entity type ‘P (Condition and disease)’, was the most common entity closely related to the P class, appearing in 18 included publications.

A notable trend within the latest review update was 23 publications now annotating or working with datasets that differentiated between intervention and control arms; fourteen of these published during or after 2022. This trend can be attributed towards relation extraction and summarisation tasks requiring this type of data. It is still common for I and C being merged for straightforward entity or sentence extraction (n=71, 61%). Most data extraction approaches focused on recognising instances of entity or sentence classes, and a small number of publications went one step further to normalise to actual concepts and including data sources such as UMLS (Unified Medical Language System). ^{35 , 39 , 59 , 73 , 85}

The ‘Other’ category includes some more detailed drug annotations ⁶⁵ or information such as confounders ⁴⁹ and other entity types (see the full dataset in Underlying data for more information ¹²⁷ ).

3.3.1 Granularity of data extraction

A total of 86 publications (73%) extracted at least one type of information at the entity level, while 59 publications (50%) used sentence level (see Table A1 extended version in Underlying data ¹²⁷ ). We defined the entity level as any number of words that is shorter than a whole sentence, e.g., noun-phrases or other chunked text. Data types such as P, IC, or O commonly appeared to be extracted on both entity and sentence level, whereas ‘N’, the number of people participating in a study, was commonly extracted on entity level only.

3.3.2 Type of input

The majority of publications and benchmark corpora mentioned MEDLINE, via PubMed, as the data source for text. Text files (n = 99), next to XML (n = 12), or HTML (n = 3), are the most common format of the data downloaded from these sources. Therefore, most systems described using, or were assumed to use, text files as input data. Ten included publications described using PDF files as input. ^{44 , 46 , 59 , 68 , 75 , 81 , 86 , 158 , 161 , 173}

3.3.3 Type of output

An increasing number of publications described structured summaries as output of their extracted data (n = 20, increasing trend between LSR updates). Alternatives to exporting structured summaries were JSON (n = 4), XML, and HTML (n = 2 each). Three publications mentioned structured data outputs in the form of an ontology or knowledge graph. ^{51 , 88 , 137} Most publications mentioned only classification scores without specifying an output type. In these cases, we assumed that the output could be saved as text files, for example as entity token/span annotations or lists of sentences (n = 102).

In the base-review we used a list of 17 items to investigate reproducibility, transparency, description of testing, data availability, and internal and external validity of the approaches in each publication. The maximum and minimum number of items that were positively rated were 16 and 1, respectively, with a median of 10 (see Table A1 in Underlying data). ¹²⁷ Scores were added up and calculated based on the data provided in Appendix A and D (see Underlying data), ¹²⁷ using the sum and median functions integrated in Excel. Publications from recent years up to 2021 showed a trend towards more complete and clear reporting.

3.4.1 Reproducibility

3.4.1.1 Are the sources for training/testing data reported?

Of the included publications in the base-review, 50 out of 53 (94%) clearly stated the sources of their data used for training and evaluation. MEDLINE was the most popular source of data, with abstracts usually described as being retrieved via searches on PubMed, or full texts from PubMed Central. A small number of publications described using text from specific journals such as PLoS Clinical Trials, New England Journal of Medicine, The Lancet, or BMJ. ^{56 , 83} Texts and metadata from Cochrane, either provided in full or retrieved via PubMed, were used in five publications. ^{57 , 59 , 68 , 75 , 86} Corpora such as the ebm-nlp dataset, ⁵⁵ or PubMed-PICO ⁵⁴ are available for direct download. Publications published in recent years are increasingly reporting that they are using these benchmark datasets rather than creating and annotating their own corpora (see 4 for more details).

3.4.1.2 If pre-processing techniques were applied to the data, are they described?

Of the included publications in the base-review, 47 out of 53 (89%) reported processing the textual data before applying/training algorithms for data extraction. Different types of pre-processing, with representative examples for usage and implementation, are listed in Table 1 below.

After the publication of the base-review, transformer models such as BERT became dominant in the literature (see Figure 3). With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lower-casing. ^{14 , 31} LLM-based methods that do not require pre-processing emerged during the 2024 update. We are therefore not going to update this table in this, or any future iterations of this LSR. We leave it for reference to publications that may still use these methods in the future.

3.4.2 Transparency of methods

3.4.2.1 Is there a description of the algorithms used?

Figure 7 shows that 43 out of 53 publications in the base-review (81%) provided descriptions of their data extraction algorithm. In the case of machine learning and neural networks, we looked for a description of hyperparameters and feature generation, and for the details of implementation (e.g. the machine-learning framework). Hyperparameters were rarely described in full, but if the framework (e.g., Scikit-learn, Mallet, or Weka) was given, in addition to a description of implementation and important parameters for each classifier, then we rated the algorithm as fully described. For rule-based methods we looked for a description of how rules were derived, and for a list of full or representative rules given as examples. Where multiple data extraction approaches were described, we gave a positive rating if the best-performing approach was described.

3.4.2.2 Is there a description of the dataset used and of its characteristics?

Of the included publications in the review updates, 109 out of 117 (93%) provided descriptions of their dataset and its characteristics. The decrease from 97% during the last review update can be attributed to a shared task with an unclear description of the dataset, as well as publications adapting existing benchmark datasets and not providing updated information.

Most publications provided descriptions of the dataset(s) used for training and evaluation. The size of each dataset, as well as the frequencies of classes within the data, were transparent and described for most included publications. All dataset citations, along with a short description and availability of the data, are shown in Table 4.

3.4.2.3 Is there a description of the hardware used?

Most included publications in the base-review did not report their hardware specifications, though five publications (9%) did. One, for example, applied their system to new, unlabeled data and reported that classifying the whole of PubMed takes around 20 hours using a graphics processing unit (GPU). ⁶⁹ In another example, the authors reported using Google Colab GPUs, along with estimates of computing time for different training settings. ⁹⁵ In the 2024 update, one LLM publication described using the OpenAI Batch API to process 682,000 RCT abstracts from PubMed, costing $3390 and requiring <3 hours. ¹⁶⁴

3.4.2.4 Is the source code available?

Figure 8 shows that most of the included publications did not provide any source code. Currently, 49 (42%) of all included publications included links to code, two additional publications provided model weights or selected parts of the code. There was a very strong trend towards better code-availabilty in the publications between base=review and the first update (n=19 published code, 83% of the new publications provided code). For the current review update, the open-source trend still continued, but has weakened due to LLM-based methods such as zero-shot prompting not requiring classic programming. We did count LLM-based publications as having provided code if they provided prompts and parameters. However, 7 out of 13 LLM publications that relied on zero-shot prompting did not provide sufficient information. Publications that did provide the source code were exclusively published or last updated in the last seven years. GitHub is the most popular platform for making code accessible. Some publications also provided links to notebooks on Google Colab, which is a cloud-based platform to develop and execute code online. Two publications provided access to parts of the code, or access was restricted. A full list of code repositories from the included publications is available in Table 2.

3.4.3 Testing

3.4.3.1 Is there a justification/an explanation of the model assessment?

Of the included publications in the base-review, 47 out of 53 (89%) gave a detailed assessment of their data extraction algorithms. We rated this item as negative if only the performance scores were given, i.e., if no error analysis was performed and no explanations or examples were given to illustrate model performance. In most publications a brief error analysis was common, for example discussions on representative examples for false negatives and false positives, ⁴⁷ major error sources ⁹⁰ or highlighting errors with respect to every entity class. ⁷⁶ Both Refs. 52, 53 used structured and unstructured abstracts, and therefore discussed the implications of unstructured text data for classification scores.

A small number of publications did a real-life assessment, where the data extraction algorithm was applied to different, unlabelled, and often much larger datasets, tested while conducting actual systematic reviews, or evaluated in other practical scenarios. ^{46 , 58 , 63 , 69 , 48 , 95 , 101 , 102 , 150 , 178}

3.4.3.2 Are basic metrics reported (true/false positives and negatives)?

Figure 9 shows the extent to which all raw basic metrics, such as true-positives, were reported in the included publications in the LSR update. In most publications (n = 99) these basic metrics are not reported, and there is a trend between base-review and this update towards not reporting these anymore. However, basic metrics could be obtained since many new included publications made source code available and used publicly available datasets. When dealing with entity-level data extraction it can be challenging to define the quantity of true negative entities. This is true especially if entities are labelled and extracted based on text chunks, because there can be many combinations of phrases and tokens that constitute an entity. ⁴⁷ This problem was solved in more recent publications by conducting a token-based evaluation that computes scores across every single token, hence gaining the ability to score partial matches for multi-word entities. ⁵⁵

3.4.3.3 Does the assessment include any information about trade-offs between recall or precision (also known as sensitivity and positive predictive value)?

Of the included publications in the base-review, 17 out of 53 (32%) described trade-offs or provided plots or tables showing the development of evaluation scores if certain parameters were altered or relaxed. Recall (i.e., sensitivity) is often described as the most important metric for systematic review automation tasks, as it is a methodological demand that systematic reviews do not exclude any eligible data.

References 56 and 76 showed how the decision of extracting the top two or N predictions impacts the evaluation scores, for example precision or recall. Reference 102 shows precision-recall plots for different classification thresholds. Reference 72 shows four cut-offs, whereas Ref. 95 shows different probability thresholds for their classifier, and describe the impacts of this on precision, recall, and F1 curves.

Some machine-learning architectures need to convert text into features before performing classification. A feature can be, for example, the number of times that a certain word occurs, or the length of an abstract. The number of features used, e.g. for CRF algorithms, which was given in multiple publications, ⁹² together with a discussion of classifiers that should be used in high recall is needed. ^{42 , 103} show ROC curves quantifying the amount of training data and its impact on the scores.

3.4.4 Availability of the final model or tool

3.4.4.1 Can we obtain a runnable version of the software based on the information in the publication?

Compiling and testing code from every publication is outside the scope of this review. Instead, in Figure 10 and Table 3 we recorded the publications where a (web) interface or finished application was available. Counting RobotReviewer and Trialstreamer as separate projects, 11 (9%) of the included publications had an application associated with it, but only 7 are usable via web-apps. Applications were available as open-source, completely free, or free basic versions with optional features that can be purchased or subscribed to.

3.4.4.2 Persistence: Can data be retrieved based on the information given in the publication?

We observed an increasing trend of dataset availability and publications re-using benchmark corpora. Only seven of the included publications in the base-review (13%) made their datasets publicly available, out of the 36 unique corpora found then. At the previous update we found 55 publications with unique corpora, with 23 available online. 40 Publications reported using one or more of these datasets in the previous version of this LSR.

For the 2024 update we again observed wide adoption of benchmark datasets but also usage of adapted and re-labeled versions of the benchmarks. In total, we found 76 publications mentioning unique datasets. Of these, 33 publications provide links to access the datasets. These datasets were then mentioned by 63 downstream papers included in this review. These number may seem high, but can be explained by many publications employing more than one dataset for validation. Table 4 shows a summary of the corpora, their size, classes, links to the datasets, and cross-reference to known publications re-using each dataset. For the base review, we collected the corpora, provide a central link to all datasets, and planned to add datasets as they become available during the life span of this living review (see Underlying data ^{127 , 128} below). Due to the increased number of available corpora we stopped downloading the data and provide links instead. When a dataset is made freely available without barriers (i.e., direct downloads of text and labels), then any researcher can re-use the data and publish results from different models, which become comparable to one another. Copyright issues surrounding data sharing were noted by Ref. 75, therefore they shared the gold-standard annotations used as training or evaluation data and information on how to obtain the texts.

3.4.4.3 Is the use of third-party frameworks reported and are they accessible?

Of the included publications in the base-review, 47 out of 53 (88%) described using at least one third-party framework for their data extraction systems. The following list is likely to be incomplete, due to non-available code and incomplete reporting in the included publications. Most commonly, there was a description of machine-learning toolkits (Mallet, N = 12; Weka, N = 6; tensorflow, N = 5; scikit-learn, N = 3). Natural language processing toolkits such as Stanford parser/CoreNLP (N = 12) or NLTK (N = 3), were also commonly reported for the pre-processing and dependency parsing steps within publications. The MetaMap tool was used in nine publications, and the GENIA tagger in four. For the complete list of frameworks please see Appendix A and D in Underlying data. ¹²⁷

3.4.5 Internal and external validity of the model

3.4.5.1 Does the dataset or assessment measure provide a possibility to compare to other tools in the same domain?

With this item we aimed to assess publications to see if the evaluation results from models are comparable with the results from other models. Ideally, a publication would have reported the results of another classification model on the same dataset, either by re-implementing the model themselves ⁹⁶ or by describing results of other models when using benchmark datasets. ⁶⁴ This was rarely the case for the publications in the base-review, as most datasets were curated and used in single publications only. However, the re-use of benchmark corpora increased with the publications in the LSR updates, where we found 63 publications that report results on one of the previously published benchmark datasets (see Table 4).

Addtionally, in the base-review, in 40 publications (75%) data were well described, and they utilised commonly used entities and common assessment metrics, such as precision, recall, and F1-scores, leading to a limited comparability of results. In these cases, the comparability is limited because those publications used different data sets, which can influence the difficulty of the data extraction task and lead to better results within for example structured datasets or topic-specific datasets.

3.4.5.2 Are explanations for the influence of both visible and hidden variables in the dataset given?

This item relates only to publications using machine learning or neural networks. Rule-based classification systems (N = 8, 15% reporting rule-base as sole approach) are not applicable to this item, because the rules leading to decisions are intentionally chosen by the creators of the system and are therefore always visible.

Ten publications in the base-review (19%) discussed hidden variables. ⁸³ discussed that the identification of the treatment group entity yielded the best results. However, when neither the words ‘group’ nor ‘arm’ were present in the text then the system had problems with identifying the entity. ‘Trigger tokens’ ¹⁰⁴ and the influence of common phrases were also described by Ref. 68, the latter showed that their system was able to yield some positive classifications in the absence of common phrases. ¹⁰³ went a step further and provided a table with words that had the most impact on the prediction of each class. ⁵⁷ describes removing sentence headings in structured abstracts in order to avoid creating a system biased towards common terms, while Ref. 90 discussed abbreviations and grammar as factors influencing the results. Length of input text ⁵⁹ and position of a sentence within a paragraph or abstract, e.g. up to 10% lower classification scores for certain sentence combinations in unstructured abstracts, were shown in several publications. ^{46 , 66 , 102}

3.4.5.3 Is the process of avoiding overfitting or underfitting described?

‘Overfitted’ is a term used to describe a system that shows particularly good evaluation results on a specific dataset because it has learned to classify noise and other intrinsic variations in the data as part of its model. ¹⁰⁵

Of the included publications in the base-review, 33 out of 53 (62%) reported that they used methods to avoid overfitting. Eight (15%) of all publications reported rule-based classification as their only approach, allowing them to not be susceptible to overfitting by machine learning.

Furthermore, 28 publications reported cross-validation to avoid overfitting. Mostly these classifiers were in the domain of machine-learning, e.g. SVMs. Most commonly, 10 folds were used (N = 15), but depending on the size of evaluation corpora, 3, 6, 5 or 15 folds were also described. Two publications ^{55 , 85} cautioned that cross-validation with a high amount of folds (e.g. 10) causes high variance in evaluation results when using small datasets such as NICTA-PIBOSO. One publication ¹⁰⁴ stratified folds by class in order to avoid this variance in evaluation results in a fold which is caused by a sparsity of positive instances.

Publications in the neural and deep-learning domain described approaches such as early stopping, dropout, L2-regularisation, or weight decay. ^{59 , 96 , 106} Some publications did not specifically discuss overfitting in the text, but their open-source code indicated that the latter techniques were used. ^{55 , 75}

3.4.5.4 Is the process of splitting training from validation data described?

Random allocation to treatment groups is an important item when assessing bias in RCTs, because selective allocation can lead to baseline differences. ¹ Similarly the process of splitting a dataset randomly, or in a stratified manner, into training (or rule-crafting) and test data is important when constructing classifiers and intelligent systems. ¹¹⁷

All included publications in the base-review gave indications of how different train and evaluation datasets were obtained. Most commonly there was one dataset and the splitting ratio which indicated that splits were random. This information was provided in 36 publications (68%).

For publications mentioning cross-validation (N = 28, 53%) we assumed that splits were random. The ratio of splitting (e.g. 80:20 for training and test data) was clear in the cross-validation cases and was described in the remainder of publications.

It was also common for publications to use completely different datasets, or multiple iterations of splitting, training and testing (N = 13, 24%). For example Ref. 56 used cross-validation to train and evaluate their model, and then used an additional corpus after the cross-validation process. Similarly Ref. 59, used 60:40 train/test splits, but then created an additional corpus of 88 documents to further validate the model’s performance on previously unseen data.

Within publications from the 2024 update, specifically with LLM-related methods employing zero or few-shot classification, we observed a reduction of transparency with respect to reporting usage of separate datasets for prompt development and testing. Often it was not clearly described how many and which texts were used for prompt development, how they were selected, and if predictions on them were included in the evaluation results. As mentioned previously, the availability of code and datasets was equally lower within the cohort of papers that employed prompt-based extraction. For papers reporting training or some form of weight adjustment on LLMs we observed reporting that adhered to good standard of practice.

3.4.5.5 Is the model’s adaptability to different formats and/or environments beyond training and testing data described?

For this item we aimed to find out how many of the included publications in the base-review tested their data extraction algorithms on different datasets. A limitation often noted in the literature was that gold-standard annotators have varying styles and preferences, and that datasets were small and limited to a specific literature search. Evaluating a model on multiple independent datasets provides the possibility of quantifying how well data can be extracted across domains and how flexible a model is in real-life application with completely new data sets. Of the included publications in the base review, 19 (36%) discussed how their model performed on datasets with characteristics that were different to those used for training and testing, and in the latest review update we found that uptake of publicly available datasets increased further. In some instances, however, this evaluation was qualitative where the models were applied to large unlabelled, real-life datasets. ^{46 , 58 , 69 , 48 , 95 , 101 , 102 , 164}

3.4.6 Other

3.4.6.1 Caveats

Caveats were extracted as free text. Included publications (N = 64, 86%) reported a variety of caveats. After extraction we structured them into six different domains: 1.

Label-quality and inter-annotator disagreements

These are further discussed in the ‘Discussion’ section of this living review.

3.4.6.2 Sources of funding and conflict of interest

Figure 11 shows that most of the included publications in the base review did not declare any conflict of interest. This is true for most publications published before 2010, and about 50% of the literature published in more recent years. However, sources of funding were declared more commonly, with 69% of all publications including statements for this item. This reflects a trend of more complete reporting in more recent years.

4.1.1 System architectures

Systems described within the included publications are evolving over time. Non-machine-learning data extraction via rule-base and API is one of the earliest and most frequently used approaches. Various classical machine-learning classifiers such as naïve Bayes and SVMs are very common in the literature published between 2005-2018. Up until 2020 there was a trend towards word embeddings and neural networks such as LSTMs. Between 2020 and 2022 we observed a trend towards transformers, especially the BERT, RoBERTa and ELECTRA architectures pre-trained on biomedical or scientific text. From 2023 onwards, the number of included publications rose sharply due to the adoption of LLMs. Zero-shot prompt-based methods created opportunities for anyone (with or without programming skills) to automate data extraction without the need to curate training data. These LLM extractions tend to be generative summaries of the data of interest, rather than exhaustive verbatim extraction of each entity of interest. 17 papers investigated LLMs, including 6 for fine-tuning, 13 for zero-shot and 2 for k-shot (reference cited in results section).

4.1.2 Evaluation

We found that precision, recall, and F1 were used as evaluation metrics in most publications, although sometimes these metrics were adapted or relaxed in order to account for partial or similar matches. Due to the generative nature of LLMs, the evaluation of zero or k-shot prompting reported in the review update diverges from previous good practice. In LLM publications, evaluators gravitated towards assessing document-level accuracy of predictions or scores such as Rouge that were initially developed for translation and generative tasks that do not align with automated data extraction. Additionally, the generative nature of LLM output had a negative effect on evaluation dataset size, because LLM evaluating requires humans for assessment (an issue that is not present with transformer, machine-learning, or rule-based systems as these can be evaluated automatically using benchmark datasets).

4.1.3 Scope

Most of the included publications focused on extracting data from titles and abstracts. The reasons for this include the availability of data and ease of access, as well as the high coverage of information and the availability of structured abstracts that can automatically derive labelled training data. A much smaller number of the included publications extracted data from full texts. Half of the 30 systems that extract data from full text were published within the last four years. In systematic review practice, manually extracting data from abstracts is quicker and easier than manually extracting data from full texts. Therefore, the potential time-saving and utility of full text data extraction is much higher because more time can be saved by automation and it provides automation that more closely reflects the work done by systematic reviewers in practice. The data extraction literature on full text is increasing but still a minority, possibly due to a lack of public benchmarking corpora as authors are concerned about copyright. Extraction from abstracts may be of limited value to reviewers in practice because it carries the risk of missing information.

4.1.4 Target texts

Reports of randomised controlled trials were the most common texts used for data extraction. Evidence concerning data extraction from other study types was less common and is discussed further in the following sections.

We only assessed full quality of reporting in the base-review, and assessed selected items during the review updates. The quality of reporting in the included studies in the base-review was found to be improving over time. We assessed the included publications in the base-review based on a list of 17 items in the domains of reproducibility, transparency, description of testing, data availability, and internal and external validity.

Base-review: Reproducibility was high throughout, with information about sources of training and evaluation data reported in 94% of all publications and pre-processing described in 89%. In terms of transparency, 81% of the publications provided a clear description of their algorithm, 94% described the characteristics of their datasets, but only 9% mentioned hardware specifications or feasibility of using their algorithm on large real-world datasets such as PubMed. Testing of the systems was generally described, 89% gave a detailed assessment of their algorithms. Trade-offs between precision and recall were discussed in 32%. A total of 88% of the publications described using at least one accessible third-party framework for their data extraction system. Internal and external validity of each model was assessed based on its comparability to other tools (75%), assessment of visible and hidden variables in the data (19%), avoiding overfitting (62%, not applicable to non-machine learning systems) descriptions of splitting training from validation data (100%).

Review updates: In terms of data availability, source code was often shared in the publications added in the LSR updates. In the base-review (which included publications up to 2021), only 15% of all includes had made their code available. After the LSR updates, 42% (N=49) now have their code available and all links to code repositories are shown in Table 2.

For testing, basic metrics were reported in only 15% (N=18) of the included publications, which is a downward trend from 24% in the base-review. However, more complete reporting of source-code and public datasets still leads to increased transparency and comparability.

Availability of the final models as end-user tools remains poor. Eleven (9%) of the included publications had an application with user-interface associated with it, but only 7 tools are deployed and directly usable via web-apps (see Table 3 for links). It is noteworthy, however, that four out of seven available tools are searchable databases with pre-extracted entities that aim to add value to reference searching. Their content is often limited to references in PubMed. These tools are designed for search and not the actual data extraction process within a literature review. There are two main drawbacks of these tools for data extraction practice. Firstly, the user is likely to obtain additional (non PubMed) references that require data extraction on demand, and these tools do not support on-demand inference. Secondly, depending on the data type that is being extracted, data then need to be added to exportable hierarchical forms or study characteristics tables, which is currently not supported by these tools. Is unclear how many of the other tools described in the literature are used in practice, even if only used internally within their author’s research groups.

There was a surprisingly strong trend towards sharing and re-using already published corpora in the LSR updates. In the base-review, labelled training and evaluation data were available from 13% of the publications. After the latest LSR update we identified 76 publications with unique corpora, 33 corpora were available online and at least 63 other included publication mention using them. Table 4 provides the sources of all corpora and publications using them, including adaptations or extensions to datasets. The most commonly used dataset for entity recognition is EBM-NLP, also referred to as EBM PICO. It is reported to be used by 28 downstream publications: 19 usages as-is, 7 making or using an adapted or extended version of it, and two publications using the Microsoft BLURB-Benchmark ¹⁴¹ [ 2], which includes EBM-NLP. For sentence classification, NICTA-PIBOSO and PubMedPICO lead with 11 and 8 publications respectively re-using their dataset. For relation extraction, EvidenceInference 2.0 is used by at least four other publications, while two additional publications extended the dataset.

We collected information on whether authors evaluated the adaptability of their algorithms by testing them on additional datasets with different characteristics, e.g. with references from a different disease domain, study type, or on a large unlabeled corpus. It is impossible (although very desirable) to quantify how well data extraction would work on real-world projects and if it performs better or worse on domain-specific data. Testing on multiple corpora with different characteristics can help with estimating how much the performance could vary when adopted into practice. An example for this is Witte 2024 ¹⁷⁶ who report that their F1-score is higher on glaucoma studies compared with diabetes 2 studies (0.63 vs. 0.54) but this has also been shown by others. ¹⁷⁹ There is a positive trend with authors of included publication increasingly using multiple corpora for evaluation, which is also aided by the availability of multiple benchmarking corpora in entity, sentence, and relation classification (see Table 4). In the base-review, adaptability was reported for 19 publications (36%), while now it is reported by 55 (47%).

Caveats and limitations noted in the included publications are discussed in the following section.

In the following section we discuss caveats and challenges highlighted by the authors of the included publications. We found a variety of topics discussed in these publications and summarised them under seven different domains. Due to the increasing trend of relation-extraction we now summarise any challenges or caveats related to these within the updated text at the end of each applicable domain and added a new section specifically focusing on LLMs.

4.3.1 Label-quality and inter-annotator disagreements

The quality of labels in annotated datasets was identified as a problem by several authors. The length of the entity being annotated, for example O or P entities, often caused disagreements between annotators. ^{46 , 48 , 58 , 69 , 95 , 101 , 102} We created an example in Figure 12, which shows two potentially correct, but nevertheless different annotations on the same sentence.

Similar disagreements, ^{65 , 85 , 104} along with missed annotations, ⁷² are time-intensive to reconciliate ⁹⁷ and make the scores less reliable. ⁹⁵ As examples of this, two publications observed that their system performed worse on classes with high disagreement ^{75 , 104} and one discussed boundary errors caused by different annotation styles between corpora. ¹³⁵ There exist different explanations for worse performance in these cases. It is possibly harder for models to learn from labelled data with systematic differences within. Another reason is that the model learns predictions based on one annotation style and therefore artificial errors are produced when evaluated against differently labelled data, or that the annotation task itself is naturally harder in cases with high inter-annotator disagreement, and therefore lower performance from the models might be explainable. An overview of the included publications discussing this, together with their inter-annotator disagreement scores, is given in Table 5.

To mitigate these problems, careful training and guides for expert annotators are needed. ^{58 , 77} For example, information should be provided on whether multiple basic entities or one longer entity annotation are preferred. ⁸⁵ Crowd-sourced annotations can contain noisy or incorrect information and have low interrater reliability. However, they can be aggregated to improve quality. ⁵⁵ In recent publications, partial entity matches (i.e., token-wise evaluation) downstream were generally favoured above complete detection, which helps to mitigate this problem’s impact on final evaluation scores. ^{55 , 83}

For automatically labelled or distantly supervised data, label quality is generally lower. This is primarily caused by incomplete annotation due to missing headings, or by ambiguity in sentence data, which is discussed as part of the next domain. ^{44 , 57 , 103}

4.3.2 Ambiguity

The most common source of ambiguity in labels described in the included publications is associated with automatically labelled sentence-level data. Examples of this are sentences that could belong to multiple categories, e.g., those that should have both ‘P’ and an ‘I’ label, or sentences that were assigned to the class ‘other’ while containing PICO information (Refs. 54, 95, 96, among others). Ambiguity was also discussed with respect to intervention terms ⁷⁶ or when distinguishing between ‘control’ and ‘intervention’ arms. ⁴⁶ When using, or mapping to UMLS concepts, ambiguity was discussed in Refs. 41, 52, 72.

At the text level, ambiguity around the meaning of specific wordings was discussed as a challenge, e.g., the word 'concentration' can be a quantitative measure or a mental concept. ⁴¹ Numbers were also described as challenging due to ambiguity, because they can refer to the total number of participants, number per arm of a trial, or can just refer to an outcome-related number. ^{84 , 113} When classifying participants, the P entity or sentence is often overloaded because it includes too much information on different, smaller, entities within it, such as age, gender, or diagnosis. ⁸⁹

Ambiguity in relation-extraction can include cases where interventions and comparators are classified separately in a trial with more than two arms, thus leading to an increased complexity in correctly grouping and extracting data for each separate comparison.

4.3.3 Variations in text

Variations in natural language, wording, or grammar were identified as challenges in many references that looked closer at the texts within their corpora. Such variation may arise when describing entities or sentences (e.g., Refs. 48, 79, 97) or may reflect idiosyncrasies specific to one data source, e.g., the position of entities in a specific journal. ⁴⁶ In particular, different styles or expressions were noted as caveats in rule-based systems. ^{42 , 48 , 80}

There is considerable variation in how an entity is reported, for example between intervention types (drugs, therapies, routes of application) ⁵⁶ or in outcome measures. ⁴⁶ In particular, variations in style between structured and unstructured abstracts ^{65 , 78} and the description lengths and detail ^{59 , 79} can cause inconsistent results in the data extraction, for example by not detecting information correctly or extracting unexpected information. Complex sentence structure was mentioned as a caveat especially for rule-based systems. ⁸⁰ An example of a complex structure is when more than one entity is described (e.g., Refs. 93, 102) or when entities such as ‘I’ and ‘O’ are mentioned close to each other. ⁵⁷ Finally, different names for the same entity within an abstract are a potential source of problems, ⁸⁴ which for example makes the extraction of outcomes challenging. ¹⁶⁴ When using non-English texts, such as Spanish articles, it was noted that mandatory translation of titles can lead to spelling mistakes and translation errors ³⁵ and that it is unknown how current algorithms perform react to non-English text. ¹⁶⁴ For the 2024 update we identified four publications describing automation on Chinese texts, one working with a traditional Chinese medicine corpus ¹⁵⁶ and three submissions to the CHIP 2023 Shared task 5: Medical Literature PICOS Identification ¹⁸² but it is unclear how well these corpora represent Chinese literature and how comparable the results are to English PICO extraction. ¹⁵⁶

Another common variation in text was implied information. For example, rather than stating dosage specifically, a trial text might report dosages of ‘10 or 20 mg’, where the ‘mg’ unit is implied for the number 10, making it a ‘dosage’ entity. ^{46 , 48 , 90} Implied information also applies when extracting the number of participants at various stages of a trial, when numbers of participants per arm need to be added in order to infer the total N or when participants are lost to follow-up. ^{157 , 158} This issue can cause the number of participants or the number of events to be inflated. ¹⁷⁴ Hoang 2022 discuss that missing information led annotators to imply information, which resulted in less consistent annotations for their gold standard, which may then in turn negatively affect the models trained on such data. ¹⁴³

Implied information was also mentioned as problem in the field of relation-extraction, with Nye et al. (2021) ⁶³ discussing importance of correctly matching and resolving intervention arm names that only imply which intervention was used. Examples are using ‘Group 1’ instead of referring to the actual intervention name, or implying effects across a group of outcomes, such as all adverse events. ⁶³

4.3.4 Domain adaptation and comparability

Because of the wide variation across medical domains, there is no guarantee that a data extraction system developed on one dataset automatically adapts to produce reliable results across different datasets relating to other domains. The hyperparameter configuration or rule-base used to conceive a system may not retrieve comparable results in a different medical domain. ^{40 , 68} Therefore, scores might not be similar between different datasets, especially for rule-based classifiers, ⁸⁰ when datasets are small, ^{35 , 49} when structure and distribution of class of interest varies, ⁴⁰ or when the annotation guidelines vary. ⁸⁵ A model for outcome detection, for example, might learn to be biased towards outcomes frequently appearing in a certain domain, such as chemotherapy-related outcomes for cancer literature or it might favour to detect outcomes more frequent in older trial texts if the underlying training data are older or outdated. ⁷³ A model trained on common RCT texts might fail to detect entities in crossover or factorial trials. ¹⁵⁰ Another caveat mentioned by Refs. 59, 85 is that the size of the label space must be considered when comparing scores, as models that normalise to specific concepts rather than detecting entities tend to have lower precision, recall, and F1 scores.

Comparability between models might be further decreased by comparing results between publications that use relaxed vs. strict evaluation approaches for token-based evaluation, ³⁴ or publications that use the same dataset but with different random seeds to split training and testing data. ^{33 , 118}

Therefore, several publications discuss that a larger amount of benchmarking datasets with standardised splits for train, development, and evaluation datasets and standardised evaluation scripts could increase the comparability between published systems. ^{46 , 92 , 114}

4.3.5 Computational or system architecture implications

Computational cost and scalability were described in two publications. ^{53 , 114} Problems within the system, e.g., encoding ⁹⁷ or PDF extraction errors ⁷⁵ lead to problems downstream and ultimately result in bias, favouring articles from big publishers with better formatted data. ⁷⁵ Similarly, grammar and parsing part-of-speech and/or chunking errors (Refs. 76, 80, 90, among others) or faulty parse-trees ⁷⁸ can reduce a system’s performance if it relies on access to correct grammatical structure. In terms of system evaluation, 10-fold cross-validation causes high variance in results when using small datasets such as NICTA-PIBOSO, ^{54 , 85 , 104} described that the same problem needs to be addressed through stratification of the positive instances of each class within folds. LLMs such as GPT-4 are commonly accessed via third-party APIs because their size and computational power requirements exceed the capacity of most home or office computers. When applying them to large datasets, such as all PubMed RCTs, methods such as employing batch-APIs to reduce time and costs were reported. ¹⁶⁴

4.3.6 Missing information in text or knowledge base

Information in text can be incomplete. ¹¹⁴ For example, the number of patients in a study might not be explicitly reported, ⁷⁶ or abstracts lacking information about study design and methods can appear, especially in unstructured abstracts and older trial texts. ^{91 , 96} In some cases, abstracts can be missing entirely. These problems can sometimes be solved by considering the use of full texts as input. ^{71 , 87}

Where a model relies on features, e.g., MetaMap, then missing UMLS coverage causes errors. ^{72 , 76} This also applies to models like CNNs that assign specific concepts, where unseen entities are not defined in the output label space. ⁵⁹

In terms of automatic summarisation and relation extraction it was also cautioned that relying on abstracts will lead to a low sensitivity of retrieved information, as not all information of interest may be reported in sufficient detail to allow comprehensive summaries or statements about relationships between interventions and outcomes to be made. ^{60 , 63}

The neural networks or machine-learning models from publications included in this review learn to classify and extract data by adjusting numerical weights and by applying mathematical functions to these sets of weights. The decision-making process behind the classification of a sentence or an entity is therefore comparable with a black box, because it is very hard to comprehend how, or why the model made its predictions. A recent comment published in Nature has called for a more in-depth analysis and explanation of the decision-making process within neural networks. ¹¹⁷ Ultimately, hidden tendencies in the training data can influence the decision-making processes of a data extraction model in a non-transparent way. Many of the examples discussed in the comment are related to healthcare, but in practice there is a very limited understanding of their inherent biases despite the broad application of machine learning and neural networks. ¹¹⁷

A deeper understanding of what occurs between data entry and the point of prediction can benefit the general performance of a system, because it uncovers shortcomings in the training process. These shortcomings can be related to the composition of training data (e.g. overrepresentation or underrepresentation of groups), the general system architecture, or to other unintended tendencies in a system’s prediction. ¹¹⁹ A small number of included publications in the base-review (N = 10) discussed issues related to hidden variables as part of an extensive error analysis (see section 3.5.2). The composition of training and testing data were described in most publications, but no publication that specifically addresses the issues of interpretability or explainability was found.

There are several corpora described in the literature, many with manual gold-standard labels (see Table 4). There are still publications with custom, unshared datasets. Possible reasons for this are concerns over copyright, or malfunctioning download links from websites mentioned in older publications. Ideally, data extraction algorithms should be evaluated on different datasets in order to detect over-fitting, to test how the systems react to data from different domains and different annotators, and to enable the comparison of systems in a reliable way. As a supplement to this manuscript, we have collected links to datasets in Table 4 and encourage researchers to share their automatically or manually annotated labels and texts so that other researchers may use them for development and evaluation of new data extraction systems.

This LSR has its cut-off in a period of very high publishing activity in the field of automated data extraction – mostly due to LLMs facilitating access to automation methods, but also due to continuing interest in transformer models. Before this update, we wrote that the arrival of LLMs ‘may mark the current state of the field at the end of a challenging period of investigation, where the limitations of recent machine learning approaches have been apparent, and the automation of data extraction was quite limited.’ The performance of LLMs did not disappoint, but their usage for automated data extraction is not yet mature. We expect to see many more publications in the near future that investigate LLM hallucinations and reproducibility issues, practical comparisons with humans, and evaluation of time-savings induced by (semi) automation methods.

Harvard Dataverse: Appendix for base review. 10.7910/DVN/LNGCOQ. ¹²⁷

This project contains the following underlying data: •

Appendix_A.zip (full database with all data extraction and other fields for base review data)

Harvard Dataverse: Available datasets for SR automation. 10.7910/DVN/0XTV25. ¹²⁸

This project contains the following underlying data: •

Datasets shared by authors of the included publications (collected for base review, see table 1_long_updated.xlsx for links to code and data for other includes)

Data are available under the terms of the Creative Commons Zero “No rights reserved” data waiver (CC0 1.0 Public domain dedication).

Open Science Framework: Data Extraction Methods for Systematic Review (semi)Automation: A Living Review Protocol. https://doi.org/10.17605/OSF.IO/ECB3T. ¹⁵

This project contains the following extended data: •

Review protocol

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Harvard Dataverse: PRISMA checklist for ‘Data extraction methods for systematic review (semi)automation: A living systematic review’ https://doi.org/10.7910/DVN/LNGCOQ. ¹²⁷

Data are available under the terms of the Creative Commons Zero “No rights reserved” data waiver (CC0 1.0 Public domain dedication).

The development version of the software for automated searching is available from Github: https://github.com/mcguinlu/COVID_suicide_living .

Archived source code at time of publication: http://doi.org/10.5281/zenodo.3871366. ¹⁷

License: MIT