A roadmap for the generation of benchmarking resources for antimicrobial resistance detection using next generation sequencing

General considerations

When discussing the different challenges described in 16, rarely can an absolute “best” answer be identified for a given question; recommendations thus need to be made, taking into account the specific purpose of the benchmark resource and the fact that they may evolve with the state-of-the-art in the field.

Still, some general observations and conclusions were proposed regarding difficulties like what type of AMR mechanisms to include, which pathogens to consider, lack of expertise and of harmonisation, rapid evolution of the fields of informatics and bioinformatics in parallel to progress of scientific knowledge on AMR. They are summarised in this section and, by discussing two main use cases for benchmarking resources (single isolates and mixed samples), represent proposals for a way through these and other challenges with common, transparent, performance-based evaluation approaches, to be applied in different fields in a “One Health” perspective.

2.1. NGS platforms

A quick analysis of the different NGS platforms currently available and in development makes it obvious that the set of reads that they produce have very different characteristics. In addition, each platform has its strengths and weaknesses. Both the error rate (about 0.1% for Illumina (RRID:SCR_010233), 1–10% for newer technologies like from Pacific Biosciences and Oxford Nanopore Technologies (RRID:SCR_003756)) and the types of errors (miscalls, insertions or deletions, or problems of particular motifs such as homopolymer sequences) vary according to the platform used. The average length of the reads can vary from hundreds (Illumina, Ion Torrent) to thousands (PacBio, Nanopore) of base pairs^22–24.

Bioinformatics pipelines are thus usually designed to handle the output of a specific platform, often in a certain configuration. Although exceptions exist (e.g. 25,26), in the context of a benchmark resource (and independently of the question asked), we thus believe that different datasets are needed for each of the different NGS platforms, each containing reads that have a profile that matches closely with the normal output of the respective technologies. It is important, in this case, to ensure that the datasets produced for the different platforms are developed so that they do not inadvertently introduce any bias resulting in apparent difference in performance among the different platforms due to composition of the benchmark dataset (e.g. when bioinformatics pipelines analysing the output of different platforms are compared). Although the absence of bias may be hard to demonstrate a posteriori, efforts should be made to ensure that the datasets derive from strategies that are as similar as possible, for example by containing reads generated from the same input samples.

The platforms for which benchmark datasets are produced should be selected based on pragmatic considerations. Ideally, equivalent resources should be available for all technologies; however, it may be necessary to prioritize the most common platforms if resource limitations are an issue. Recent surveys have shown a clear preference for the Illumina platform in this context^27,28. The same trend can be observed when counting the number of published articles in a scientific literature database (Figure 2). That being said, sequencing technologies are constantly evolving, and benchmark datasets should be extended to new technologies with improved performance as these are increasingly adopted by testing laboratories.

Figure 2. Number of articles published each year in the scientific literature mentioning the selected platform.

Source: Scopus, using the search: ALL ("X" AND "antimicrobial resistance").

The so-called “Third Generation Sequencing” technologies that sequence single DNA molecules and, importantly, produce long reads, have been shown to provide substantial benefits in the context of AMR detection. First, many resistance genes are located on plasmids, which are challenging to assemble using short-read sequencing technologies, because the short read lengths do not allow spanning of repetitive regions²⁹. The presence of an AMR determinant on a plasmid is also important for its transfer and eventual spread, and thus their correct assembly using long-read technologies represent a substantial advantage^30–34. In addition, the proper and timely treatment of a pathogen infection is critical for successful prevention and control of diseases in clinical settings as well as in the community. In line with this, the Nanopore sequencing technology has shown the promise of providing accurate antibiotic resistance gene identification within six hours of sample acquisition^35–37 We thus propose to include DNA Nanopore sequencing as an additional priority platform to develop benchmark resources.

The choice of formats for the different components of the datasets is also important. Each instrument produces a raw data output in a specific format (for example, the Illumina platforms generate raw data files in binary base call (BCL) format, while the Nanopore platforms produce FAST5 (HDF5) files). However, the entry point of most bioinformatics pipelines in this context is the description of the sequence of the produced reads, with an indication of the quality (or confidence) score for each of the base positions. The FASTQ format is a standard format in this context³⁸, which should be used in the benchmark resources; many tools exist to convert the raw data output files into this format in case of different platform outputs (see, for example,^39,40) although, it should be noted, different tools may produce different results and this step should be carefully planned.

Other standard formats exist to describe intermediate states of processing, for example for the description of assembled contigs or variant calling⁴¹. However, using these formats would make an a priori assumption about the strategy of the bioinformatics pipeline that may not be universal; indeed, not all reported solutions involve assembling reads, or mapping them to reference genomes or databases (see, for example,^42,43).

2.2. Datasets origin

Three main sources of data for creating a benchmark dataset were identified. The first is to simulate the output (reads) in silico using an input sequence of a resistant pathogen and a specialised software. The second is to use the archived output of previously performed experiments that are available in different repositories. The third is to perform NGS experiments on biological samples.

Although the disadvantage of simulating in silico data is obvious (it is not ‘real’), there are some substantial advantages: it is a lot cheaper than performing sequencing runs, a lot faster, and can be applied to any genome previously sequenced. Thus, many more potential scenarios can be tested, for which the ground truth is well-established (i.e., the annotation of the genome reference that is used: different species, different classes of AMR, different localization of AMR), which usually cannot be done by actually sequencing them. Finally, it is also potentially ‘safer’ to do this for pathogenic bacteria for which high biosafety levels would be required to sequence in a laboratory. However, a major drawback is that simulating variation the way nature evolves is very challenging – genetic variation happens in places in the genome where it is hardest to find.

Many methods and programs have been developed to simulate genetic data. Their use in this context is, in itself, an exercise of Open Science and mechanisms should be used to guarantee quality and reproducibility (see 44) In 2013, Peng et al.⁴⁵ developed the catalogue “Genetic Simulation Resources” (GSR, available at https://popmodels.cancercontrol.cancer.gov/gsr/) to help researchers compare and choose the appropriate simulation tools for their studies. However, after reviewing the software listed in the GSR catalogue, the authors realised that the quality and usefulness of published simulation tools varied greatly due to inaccessible source code, lack of or incomplete documentation, difficulties in installation and execution, lack of support from authors and lack of program maintenance⁴⁵. For these reasons, a defined checklist of features that may benefit end users was defined⁴⁶; the “GSR Certification Program” was developed and recently implemented into the GSR in order to assess simulation tools based on these criteria⁴⁷. Established criteria are grouped to attribute four “certificates” (https://popmodels.cancercontrol.cancer.gov/gsr/certification/):

As of December 2019, the GSR catalogue lists 148 simulators and many of them have been assessed for their compliance with the requirements in order to be certified. Obviously, not all of them are for simulation of NGS reads. In 2016 Escalona et al.⁴⁸ identified and compared 23 computational tools for the simulation of NGS data and established a decision tree for the informed selection of an appropriate NGS simulation tool for the specific question at hand.

By browsing the GSR catalogue, 20 out of 23 tools assessed by Escalona et al. (45) have been recorded, including only one with the four “GSR certificates” (Table 1), i.e. the ART tool⁴⁹. Other tools not assessed by Escalona are also present in the GSR catalogue with certificates, like NEAT⁵⁰ and VISOR⁵¹.

For choice of the simulation methods and programs for NGS reads, the decision tree proposed by Escalona et al. is robust. However, it should be complemented by “certification” steps and, in this respect, we encourage the use of the “certification” criteria established by the GSR Certification Program, to tackle the challenge of following agreed principles for rigorous, reproducible, transparent, and systematic benchmarking of omics tools, in line with those proposed by Mangul et al.¹³.

Using pre-existing experiments, from private or public repositories, ensures that the components of the dataset are representative of a real-life experiment, including the complete panel of real-life variabilities that are difficult to simulate. The main issues then are: a) there is a need to demonstrate that the experiment met the necessary quality criteria (see section 2.3); b) the “correct” value (i.e. the ‘ground truth’) for the experiment needs to be determined. This can be already described in the metadata associated with the record and/or determined (verified) a posteriori – although this requires strict annotation of the experiment; c) it will not be possible (besides rare exceptions) to build datasets for the different platforms using the same initial samples.

Generating experiments specifically for the sake of a benchmark dataset has almost the same advantages and disadvantages as using pre-existing data. Additional advantages include a better capacity to determine the “ground truth” of each sample by ensuring access to the original pathogen, as well as the possibility to generate datasets for the different platforms while using the same samples, if the same pathogen/purified DNA is processed through the different protocols and instruments. This also allows better control of the quality aspects of the procedure performed, e.g. through the use of accredited laboratories who have therefore demonstrated by audits that they possess the necessary knowhow and expertise to create high-quality data. However, an additional disadvantage is that this process requires a substantial investment of time and resources (although this investment may be deemed worthwhile given the importance of the topic, and could benefit from the involvement of the instrument vendors).

Because each approach has advantages and disadvantages, the choice must be carefully considered, according to the purpose of the dataset, which will be discussed in section 3.

2.3. Quality metrics

The quality of the original sample and the wet laboratory procedures (e.g. DNA extraction, library preparation and sequencing) have a strong impact on the quality of the reads fed into the bioinformatics pipelines. Contamination, low amounts of reads passing the machine QC, higher error rates than normal, etc. can influence the output of bioinformatics pipelines. Usually, the pipelines are designed to be resilient to some extent to these variations.

Although understanding this resilience is important, we propose, as shown in Figure 1, to separate these considerations from resources meant for quality control and performance evaluation (questions 1, 2a and 2b) for two reasons: first, many of these factors are variable, heterogeneous, technology-specific, and can be implemented at different stages of the bioinformatics pipeline; attempting to incorporate them all in the same resource would be impractical and too costly. Second, pipelines implemented for regulatory or clinical decision-making will be incorporated into a larger quality assurance framework that will ensure the quality of the input until that step². Although examples exist of end-to-end NGS workflow validation (like in the case of WGS) where bioinformatics is one of the components⁵², our approach emphasises a process where each step is validated separately (see 53).

It is then crucial to closely follow the proposed quality control schemes, either published or in development, in particular for the upstream steps (DNA isolation, library extraction, sequencing, etc.), for example ISO/TC 34/SC 9/WG 25. From these, both the metrics and the thresholds that can be applied at the level of the reads should be identified (some of which may vary according to the sequencing methodology), such as percent of bases with quality scores over Q30, percent alignment and error rates of the positive control (if present), the number of reads after trimming, quality of the draft assembly (e.g. N50, number of contigs), etc. Tools exist that can provide a panel of quality metrics from FASTQ files, such as FASTQC (RRID:SCR_014583)⁵⁴. It is important to include the quality metrics as metadata in the dataset samples.

For studies evaluating resilience (question 3), a variety of datasets are needed for the “low quality dimensions” to be tested. For example, datasets incorporating high error rates, contaminating reads, reads with lower Q-scores could be used to assess resilience of pipelines to quality issues that may or may not be detectable with standard QC pipelines. For this reason, the establishment of standard datasets for this type of benchmarking is a complex exercise and answering question 3 of Figure 1 should be attempted on a case-by-case basis, and may be better suited to individual studies. One way to harmonise the approach would be to use the datasets produced for questions 1 and 2 as a starting point, as there are tools that can add some extent of “noise” to existing good quality datasets⁴⁹.

2.4. Choice of bacteria/resistance to include

In the context of challenging/evaluating a bioinformatics pipeline for the detection of AMR genetic determinants, a very pragmatic approach could be the generation of random DNA sequences, to which particular sequences of interest are added (i.e. fragments of AMR genes). However, the genomic background of the bacteria (i.e. the “non-AMR related” sequences) might have a profound influence on the performance of the pipelines. For example, pipelines that include a contig assembly step will be affected by the frequency and level of repetitive sequences in the background genome, as well as its GC content^55,56. Some species also have genes that are similar at the sequence level to known AMR determinants that efficient pipelines must be able to distinguish.

In conclusion, the bacterial species included in the benchmark datasets, and the AMR genes they contain, need thus to be carefully selected, with the appropriate justifications. These are specific to the purpose of the dataset (Figure 1) and will be discussed in section 3.1–section 3.3 below.

2.5. Genomic and phenotypic endpoints

A pipeline processing sequencing information for AMR can produce two closely linked but conceptually different outputs: a) they can detect the genetic determinants of AMR (genomic endpoint), and in addition b) some can predict the AMR/susceptibility of the bacteria in the original sample (phenotypic endpoint).

In a clinical context, the phenotypic endpoint is the most relevant, as it provides the information that is most useful for the end users. Studies that evaluated AMR genotype to phenotype relationships have indicated that despite generally high correspondence, this can vary greatly between pathogens / case studies, and even for different antimicrobial agents within the same species^57,58. There are different reasons for discrepancies between phenotype and genotype, including the extent of the expression of the resistance determinants for the resistance to be conferred, and also relatively complex molecular pathways that can influence the eventual phenotype. In some cases, genes can also confer reduced susceptibility (i.e. increasing the concentration of an antimicrobial necessary for treatment) rather than resistance per se. A genotypic endpoint may also be problematic due to the definition of “antibiotic resistance” in different settings⁵⁹, which can complicate the interpretation of results.

In practice, however, focusing on a genomic endpoint has many advantages:

2.6. Benchmark datasets metadata

Besides the set of reads themselves, additional information needs to be associated with each sample in the dataset, not for the benchmarking per se but its use for next benchmarking exercises.

Obviously, each sample needs to include a “true” value, i.e. the ‘ground truth’ to be used for comparison when evaluating the performance of the pipeline. For a genotypic endpoint, this would take the form of the name (with a reference to a database) of the AMR determinants present. If real-life samples are used, the phenotypic information should be included, on top of the genotypic endpoint.

Public resources and repositories are available to host both the data and the metadata, and should be used as appropriate for the sake of transparency, obsolescence and traceability of the datasets of the benchmark resource. In practice, this means:

3.1. Is the bioinformatics pipeline performing according to (agreed) minimum standards?

The scope of this benchmark resource is to address the questions of validation, implementation and quality control over time (i.e. following any change in the pipeline or the environment on which it is executed). The dataset required for this should be compiled based on an agreed “minimum” standard, i.e. thresholds for the acceptance values of certain performance metrics for the bioinformatics pipeline in the context of the detection of AMR determinants, no matter the exact final use of the information produced.

This evaluation of performance should be based on challenging the pipeline with input representing a carefully selected set of resistance determinants and bacterial hosts. These sets of NGS reads should be fully characterised regarding their genetic content and serve as (in silico) reference materials for the validation and quality control of the bioinformatics component of the methods (see, for other host models, 66,67).

To maintain this necessary control on the genetic content of the reads, the dataset could be composed exclusively of simulated experiments. Synthetic artificial reads can be generated on a large scale in a harmonised manner and most importantly allow full control on the content. Alternatively, real sequencing datasets could be used, which would be extremely relevant for cases where the presence/absence of some AMR determinants has been established using consolidated classical molecular-biology-based methods and/or first generation-sequencing (e.g. PCR and/or Sanger sequencing). Both approaches enable the generation of datasets for which the ‘ground truth’ is well-established.

For the choice of resistances and bacterial species to be included, it is proposed to select them based on three sources, based on their current public health relevance and regulatory frameworks:

Table 2 shows the combination of these three lists, in terms of both the bacterial species and the antibiotics mentioned.

It is important to highlight that the combination of these three lists still leaves important regulatory gaps, and should be complemented by the World Organisation for Animal Health (OIE, RRID:SCR_012759) list of antimicrobial agents of veterinary importance⁷¹ or others⁷². However, the lists do not mention specific species associated to each antibiotic, and these should be selected by the appropriate experts for the context of this benchmark resource. In practice, the generation of reference sequencing datasets should focus on:

The availability of reference datasets for which the ground truth is known, i.e. the composition of AMR determinants within the sample (whether SNPs, genes, or more complex features) is well-established, allows to compare the output of bioinformatics pipelines to evaluate their performance. The endpoint considered for this benchmark is thus genotypic (see section 2.5), i.e. the proper detection and identification of genomic AMR determinants is evaluated. The reference datasets for each pathogen should be carefully chosen and characterised to ensure all present AMR determinants are carefully recorded.

Several classical performance metrics used traditionally for method performance evaluation can then be used to describe performance of the pipeline, such as sensitivity, specificity, accuracy, precision, repeatability, and reproducibility². Because of the selection of this subset of bacteria/resistances and their immediate clinical and regulatory importance, an important performance metric to be evaluated is accuracy, i.e. the likelihood that results are correct. Definitions for accuracy and other performance metrics should be carefully considered and tailored to the genomics context⁵³ (for example, “reproducibility”, could be evaluated as the result of running the same bioinformatics pipeline, with the same datasets, implemented in different systems).

For all performance metrics, the minimum acceptable values should be subsequently determined once the outputs of real benchmarking exercises considering all the aspects described in this article are available. This will allow to enforce thresholds that pipelines should obtain. Such an approach focusing on performance-based evaluation offers a robust framework for pipeline validation that is more flexible than focusing all efforts on single pipelines or sequencing technologies. Scientists should be able to choose the pipeline best suited to their own needs, whether commercial, open-source, web-based, etc.⁷⁵, as long as it conforms to minimum agreed upon performance standards, evaluated by analyzing the community AMR benchmark datasets that are created. This also offers flexibility with respect to the currently quickly evolving sequencing technologies for which it is not always clear yet which algorithmic approaches will become community standards, allowing different data analysis methodologies if minimum performance is demonstrated.

When generated, the benchmark should be deployed on a dedicated (and sustainably maintained) platform that includes all the links to the data (see section 2.6) and a description of all the steps/decisions that were taken to generate it. It is also important to implement, from the start, a clear version control system for the benchmark resource, in order to properly document the changes over time, and the exact versions used at the different times that the resource is used. In addition to the unique accession numbers of the individual samples in their respective repositories, the dataset as a whole should have a unique identifier (e.g. a DOI) that changes when any modification is made. The versioning should also allow access to and use of any previous versions of the resource, even after being updated.

This minimal dataset contains, by definition, a limited number of species and may lack pathogens of clinical importance (for example, Mycobacterium tuberculosis, for which WGS-based approaches have shown particular advantages, see 76,77). A full validation exercise for a specific pipeline, applied to a specific context, will need additional samples that complement the resource described in this section with the appropriate species/resistances. These datasets may, for example, be taken from the resources described in the following sections, that focus on evaluating the actual performance of methods in broader contexts by gathering the many datasets necessary to do so.

3.2. What is the quality of the information produced by the bioinformatics pipeline (prediction of resistance)?

The scope of this benchmark resource is to identify gold standards for bioinformatics pipelines, in this case linked to the specific use of predicting resistance/susceptibility of a pathogen.

There is a step between identifying the determinants of AMR and predicting resistance, which is not always straightforward as factors such as expression of the AMR gene may affect the prediction^57,74. For this reason, and because it is conceptually closer to the information that is acted upon, the endpoint for this benchmark should be phenotypic. In addition, the dataset should be composed of real NGS experiments, since artefacts and variations are more complex in real sequencing reads than in simulated reads, a factor crucial to consider for this scope that focuses on accuracy.

To minimise the need of extensive resources to produce these “real” datasets, we propose to focus on re-using experiments previously performed under standardised conditions. A great source of data are the published ring trials; these have the additional advantage of providing an accurate characterisation of the sequenced samples, since the same original samples are sequenced many times by different laboratories. If needed, the data generated by single-site studies can also be evaluated, although in this case the issue of the correct characterisation of the samples (their “true” resistance patterns) should be addressed. One possibility is to use studies performed in a hospital setting, linked to clinical outcome (for example,⁷⁸), or where sufficient information is available to evaluate the way the susceptibility testing was performed.

In practice, this would mean:

These datasets would then be used to test and compare the different bioinformatics pipelines in order to calculate the accuracy of their phenotypic predictions. Although not exhaustive, these datasets should cover the most relevant “real-life” cases, as they warranted their inclusion into a ring trial, with the associated resources committed to produce the data. The final size and composition (species, resistances) of the dataset would depend on what is provided by the available projects; ad hoc ring trials could be organised to cover eventual important gaps in species and/or resistance.

Although the chosen endpoint is mostly phenotypic, the purpose is to evaluate bioinformatics pipelines that process information at the sequence level, so it was agreed that there was little added value of inserting resistant samples (based on a characterised or inferred phenotype) for which the resistance mechanism is still unknown. In any case, it is improbable that these cases would have been included in ring trials projects.

Although the performance metrics described in Section 3.1 apply and are relevant in this case, the main performance metric for this benchmark is the accuracy. Because of the difficulty of predicting the link between the presence of AMR determinants and their impact on the pathogen susceptibility to the antimicrobial agents, the target accuracy is expected to be lower than for a genotypic endpoint. Both false positives and false negatives can be an issue when the information is used for clinical intervention, so a sufficient amount of “borderline” cases should be included, and both sensitivity and specificity evaluated. It is also possible to consider attaching different relative costs for false positives and false negatives when evaluating the accuracy metrics.

Once selected and combined, the data should be separated by NGS platform, and by species and antibiotic. Because this benchmarking aims at evaluating and comparing performance of methods, which are continuously developed and optimised, against a large and constantly expanding dataset, it is crucial to define an environment where the AMR community can establish a continuous benchmarking effort. Within this platform, pipelines would be compared simultaneously based on up-to-date datasets, under the same conditions, and over time. Constantly updating and adding to the reference datasets is important both to keep up with the evolution of the knowledge/reality in the field, and to avoid that pipelines are developed that are optimised to specific datasets only.

One option is OpenEBench (Open ELIXIR Benchmarking and Technical Monitoring platform), which is developed under the ELIXIR-EXCELERATE umbrella in order to provide such a platform¹⁸. In this framework, in addition to compiling the data resources to be included (as described above), and whatever the platform chosen, there will be the need for efforts to:

Provisions should be included to allow the possibility to evaluate, in this context, pipelines that cannot be made “FAIR” based on intellectual property rights, institutional policies or available resources.

A final step will be to communicate these efforts within the scientific community and the potential end users, as well as to demonstrate the added value of this “live” benchmark resource to ensure that future studies (in particular, their pipelines and the datasets they generate) are efficiently integrated in the platform.

3.3. What is the quality of the information produced by the bioinformatics pipeline (mixed samples)?

Many gaps exist in the scientific understanding of antibiotic resistance development and transmission, making it difficult to properly advise policy makers on how to manage this risk. There is strong evidence that a multitude of resistance genes in the environment have not yet made it into pathogens^103,104; understanding the relative importance of different transmission and exposure routes for bacteria is thus crucial^59,105–107.

Establishing a baseline for resistance determinants in the environment, and linking this to a surveillance scheme, requires a good understanding of the relative performance of methods that are and have been developed to characterise the resistome in a complex sample. There would be, also for this use case, a great value in the establishment of a community-driven “live” benchmarking using a platform such as OpenEBench, and many of the concepts that were discussed in section 3.2 apply here as well, with the following differences:

The resistome is usually derived from a sample containing a complex microbial community (see 111–123 for recent examples). For this reason, the approaches¹¹⁴ and tools¹¹⁵ from the ongoing Critical Assessment of Metagenome Interpretation (CAMI) could be considered when organising the community around this challenge.

In practice, this means an effort to engage and coordinate the community of bioinformatics pipelines designed to predict the resistome of a sample in order to:

The community should decide whether (or at what stage) the use of real data can also be considered in the challenge. As for purified bacteria (see Table 3), many studies have been published as potential sources of raw data. These studies can also be used as a source of information to define the relevant profiles (point 2 above). Recent studies include resistome determination in samples from drinking water^116,117, wastewater plants^118,119, hospital wastewater^120,121, human gut^111,122, sewage¹²³, to name a few. In the benchmarking platform, the datasets (and the calculated pipeline performances) should be separated by the type of source they originate from or simulate. Another important point for this scope is the detection of minority populations and the correct normalisation of the samples to be analysed¹²⁴.