Large-scale quality assessment of prokaryotic genomes with metashot/prok-quality

Metagenomic sequencing allows large-scale identification and genomic characterization. Binning is the process of recovering genomes from complex mixtures of sequence fragments (metagenome contigs) of unknown bacteria and archaeal species. Assessing the quality of genomes recovered from metagenomes requires the use of complex pipelines involving many independent steps, often difficult to reproduce and maintain. A comprehensive, automated and easy-to-use computational workflow for the quality assessment of draft prokaryotic genomes, based on container technology, would greatly improve reproducibility and reusability of published results. We present metashot/prok-quality, a container-enabled Nextflow pipeline for quality assessment and genome dereplication. The metashot/prok-quality tool produces genome quality reports that are compliant with the Minimum Information about a Metagenome-Assembled Genome (MIMAG) standard, and can run out-of-the-box on any platform that supports Nextflow, Docker or Singularity, including computing clusters or batch infrastructures in the cloud. metashot/prok-quality is part of the metashot collection of analysis pipelines. Workflow and documentation are available under GPL3 licence on GitHub.


Introduction
Genome-resolved metagenomics is one of the most promising approaches to identify and characterize novel microbial species. Large-scale environmental and host-associated studies demonstrated how metagenomics can expand our knowledge of uncultivated prokaryotes, recovering thousands of metagenome-assembled genomes (MAGs) of new archaeal and bacterial species. 1,2 For this reason, automated and reproducible methods for assessing the quality of MAGs play a critical role.
To recover MAGs, metagenomic sequence reads are first assembled into contigs using specific algorithms. 3 Contigs are then processed by tools like MetaBAT 2 4 or VAMB 5 that use tetra-nucleotide frequency (TNF) profiles and abundance patterns to group sequences that are likely to belong to the same organism (binning). Binning improves the interpretability of metagenomic data, but at the same time represents (together with assembly) a significant source of error. 6 Manual refinement 7 can increase the quality of resulting MAGs, but undermines the reproducibility of the analysis and is unfeasible for large-scale studies.
The recently introduced Minimum Information about a Metagenome-Assembled Genome (MIMAG) standard 8 recommends a set of measures for assessing the quality of MAGs. This comprises basic assembly statistics (e.g. N50), genome completeness, contamination and the presence of ribosomal RNA (rRNA) and transfer RNA (tRNA) genes.
Recovering this information involves computational pipelines composed of a series of specialized tools that are often difficult to use and install. Moreover, each task can require parameters and custom scripts that are often poorly documented, making reproducibility of results challenging. Tools and standards such as Galaxy, 9 Nextflow 10 and the Common Workflow Language, 11 coupled with container technologies like Docker, allows researchers to circumvent these issues, providing a way to build, run and share reproducible computational workflows. 12 We present metashot/prok-quality, a comprehensive and easy-to-use Nextflow pipeline for assessing the quality of draft prokaryotic genomes. Metashot/prok-quality reports the quality statistics and estimates recommended by the MIMAG standard. Basic assembly statistics, completeness, both redundant and non-redundant contamination, rRNA and tRNA genes are reported in a single, comprehensive table.

Implementation
Metashot/prok-quality is written using the Nextflow domain-specific language. Nextflow is a framework for building scalable scientific workflows using containers, allowing implicit parallelism on a wide range of computing systems. Reproducibility is guaranteed by versioned Docker images, which enclose software applications together with their dependencies, allowing isolation from the host environment and portability across platforms. metashot/prok-quality v1.2.0 is composed of five main modules ( Figure 1) and includes several custom scripts, designed to manipulate the output of the different tasks.
Software included in version 1.2.0: CheckM v1.1.2. Several tools have been developed for the assessment of completeness and contamination of MAGs. The proposed workflow includes the widely used CheckM 13 which estimate these metrics using ubiquitous and lineagespecific, single-copy core genes (SCGs) catalogs. CheckM is also used to recover the basic assembly statistics. GUNC v1.0.1. SCG-based tools like CheckM can have very low sensitivity towards contamination by fragments from unrelated organisms (non-redundant contamination). 6 In order to circumvent this problem, the recent GUNC 14 tool was added to the pipeline. GUNC quantifies the lineage homogeneity of contigs with respect to the full gene complement, accurately detecting chimerism induced by both redundant and non-redundant contamination.
Barrnap v0.9. The presence of 5S, 23S and 16S rRNA genes is predicted by the BAsic Rapid Ribosomal RNA Predictor (Barrnap) using Hidden Markov models (HMM). Both bacteria and archaea databases are used.
tRNAscan-SE v2.0.6. tRNA genes are searched using tRNAscan-SE, 15 using bacteria and archaea covariance models. The number of tRNAs and tRNA isotypes found is reported. dRep v2.6.2. Dereplication is a procedure that groups the input genomes according to their whole-genome similarity, using metrics such as the Average Nucleotide Identity 16 (ANI). Dereplication dramatically simplifies downstream analysis when the input genomes come from different sources. 17 In the proposed workflow, filtered genomes (genomes that pass completeness, contamination and GUNC filters) are optionally dereplicated using dRep. 18 For each cluster, dRep reports, as the cluster representative, the best-scoring MAG using the CheckM's quality estimates. The score is computed using the following formula: score = completeness À 5 Â contamination + 0.5 Â log(N50) Python3 custom scripts. The workflow includes three Python3 custom scripts, designed to manipulate the output of the different steps. The scripts make use of NumPy, 17 Pandas and scikit-learn libraries.
Operation metashot/prok-quality v1.2.0 requires Docker and Nextflow (tested on v20.07.1). Alternatively, the Singularity container engine can be used in place of Docker. At least 70 GB of RAM is required, a limit imposed by CheckM (v1.1.2). The workflow can run in a workstation with 16 GB of RAM using the options --reduced_tree and --max_memory 16.GB.

Use case
As mentioned above, metagenome assembly tools combine the sequence reads into larger regions called contigs.
Recently, many metagenomic assembly tools have been proposed. Amongst these, metaSPAdes 3 and MEGAHIT 19 have been shown to be able to efficiently handle large-scale short read sequencing data, producing high-quality contigs. Metagenomics contigs are then processed by tools like MetaBAT 2 4 in order to group sequences that are likely to belong to the same organism (binning). After binning, it is essential to assess the quality of the resulting candidate draft genomes.   table); ii) a directory containing the bins filtered according the completeness and contamination thresholds; iii) a TSV file listing the cluster membership of each genome after the dereplication (optional) and iv) a directory containing the cluster representatives. The original outputs of each task (e.g. Barrnap's GFF output) are also reported in dedicated folders.
In this section, we will show how to assess the quality of draft prokaryotic genomes using metashot/prok-quality. Given a series of candidate genomes in FASTA format stored in the "bins" directory, the version 1.2.0 of the workflow can be run with the following command line: nextflow run metashot/prok-quality -r 1.2.0 \--genomes 'bins/*.fa' \--outdir results A series of files and directories are created in the output directory results. The main output file is "genome_info.tsv". This TSV file contains, for each input genome, a set of quality statistics, including completeness, contamination, GUNC filter, N50, rRNA genes found, number of tRNA and tRNA types. The columns included in this file are: • Genome: the genome filename; • Completeness, Contamination, Strain heterogeneity: CheckM estimates; • GUNC pass: if a genome does not pass GUNC analysis it means it is likely to be chimeric; • Genome size (bp), ... , # predicted genes: basic genome statistics (see https://github.com/ Ecogenomics/CheckM/wiki/Genome-Quality-Commands#qa); • 5S rRNA, 23S rRNA, 16S rRNA: "Yes" if the rRNA gene was found; • # tRNA, # tRNA types: the number of tRNA and tRNA types found, respectively.
The directory "filtered" contains the genomes (in FASTA format) filtered according to --min_completeness, --max_contamination and --gunc_filter options (see below). The TSV file "genome_info_filtered.tsv" includes the same information as "genome_info.tsv", but for the filtered genomes only. Representative (dereplicated) genomes (default ANI threshold 0.95) are reported in the "filtered_repr" folder. The companion file "derep_info.tsv" contains the summary of the dereplication procedure, including the genome filename, the cluster ID and the representativeness. A set of secondary directories contains the original output of each tool included in the pipeline: • checkm: contains the original CheckM's "qc" file; • gunc: contains the original GUNC output file; • barrnap: includes the predicted rRNA sequences for bacteria (.bac) and archaea (.arc) models in GFF and FASTA formats; • trnascan_se: includes the predicted tRNA sequences for bacteria (.bac) and archaea (.arc) models in TSV and FASTA formats; • drep: dRep original data tables, figures and log file.
The command options are:

InBioS -PhytoSYSTEMS, Eukaryotic Phylogenomics, University of Liège, Liège, Belgium
This tool is useful and works as advertised by the authors. The manuscript reads well and I only have very minor comments.

Functionality:
I tested metashot/prok-quality on a HPC facility running the SLURM grid engine. The authors' instructions were nearly enough to make it work on my first attempt. However, getting right the Singularity options was a little trickier than expected. In this regard, Docker is easier to use but generally not available in HPC environments. I contacted the authors by email (posing as a regular user) and they replied immediately, confirming that the reported issue was on my side and not a bug in their package. Unfortunately, I had to attend more pressing matters and only got back to the issue recently, hence this delayed review report. I apologize for this. For the record, here are the lines required to run the tool successfully on my system: # in prok-quality/nextflow.config singularity.enabled = true singularity.cacheDir = "$PWD" singularity.autoMounts = false singularity.runOptions = "-B /path-to-user-home/prok-quality-1.2.0_with_test_data/ -B /tmpscratch/username" The interface of metashot/prok-quality is straightforward and well designed, with attention to details, such as the possibility to deal with limited RAM (--reduced_tree option for CheckM) and to economize resources (--gunc_db option to avoid downloading the GUNC database multiple times). I only have a minor complaint: the --genomes option expects input filenames but if these do not match the --ext option (e.g., .fasta instead of .fa), infiles are silently ignored. In my view, the --ext option would make more sense with a --genomes option expecting an input directory. Indeed, when infiles are specified, there is no real need for filtering.
The tool's outputs are easy to understand and well documented. Testing it with default thresholds on a chimerical bacterial genome (Cornet and Baurain 2022, Genome Biology, in press) indeed flags it as contaminated and excludes it from the downstream dereplication step. In contrast, the two corresponding clean genomes undergo through all the steps of the pipeline, as expected.

Manuscript: Abstract:
Eukaryotic sequences can be found in metagenomic samples. These are more difficult to deal with, but should be mentioned in the second sentence.

Methods:
Some sentences are either too much or too little, for example those about the rationale for dRep. I think that one more sentence would help and it would be better to move the whole idea of dereplication at the end of the Introduction. Indeed, Methods should not explain concepts.
○ Figure 1: The figure is nice. If I had to quibble, I would say that "filtered bins" can be misleading: here, the authors mean "bins satisfying their criteria of completeness and contamination" whereas one could imagine that they mean "bins cleaned up from contaminating sequences" (i.e., bins are passed "as is" and are not modified by the pipeline). Moreover, showing on the figure the parameters controlling the thresholds would help the reader to realize that they can be user-specified. Finally, a word is missing in the legend: "according the completeness.." should read "according to the completeness..."

○
The score formula should specify if completeness and contamination metrics are computed in percentage or not. Moreover, it might be interesting to make it usertweakable (only a suggestion). ○ bioinformatics tools for non-expert users; and a portable, scalable workflow to process datasets of increasing sizes.
The code is available on github. We were able to download, install and use the workflow as advertised. We have only very minor comments, to be addressed at the authors' discretion: The authors restrict their discussion to metagenome-assembled genomes. However, there is no inherent reason why the prok-quality workflow shouldn't be used for reference genomes or isolate sequencing as well, and this may be worth emphasizing. In practice, many users work on large integrated datasets of MAGs and reference genomes, where fast, consistent quality control and clustering using a workflow such as prok-quality is highly relevant.

○
In the bigger picture of an entire genome-resolved metagenomics workflow, from raw reads to biological analyses, prok-quality covers a reasonable chunk: the quality control and 'dereplication' (clustering) of genomes. That way, it can be used as a module, independently of upstream (assembly & binning) and downstream (annotation, analysis) tool choices. However, in our view, prok-quality would benefit from the addition of a taxonomic classifier, e.g. GTDB-tk. Taxonomic information would fit in very well with the reported quality metrics; but in particular, the workflow could (optionally) provide consensus taxonomies for dRep 95% ANI clusters which would greatly add value for non-expert users.  Minor: An option to switch from running with docker to running with singularity (e.g. 'profile singularity' with config profiles) would greatly enhance the ease of using this workflow as some clusters prohibit use of docker.