Jupyter notebook-based tools for building structured datasets from the Sequence Read Archive

The Sequence Read Archive (SRA) is a large public repository that stores raw next-generation sequencing data from thousands of diverse scientific investigations. Despite its promise, reuse and re-analysis of SRA data has been challenged by the heterogeneity and poor quality of the metadata that describe its biological samples. Recently, the MetaSRA project standardized these metadata by annotating each sample with terms from biomedical ontologies. In this work, we present a pair of Jupyter notebook-based tools that utilize the MetaSRA for building structured datasets from the SRA in order to facilitate secondary analyses of the SRA’s human RNA-seq data. The first tool, called the Case-Control Finder, finds suitable case and control samples for a given disease or condition where the cases and controls are matched by tissue or cell type. The second tool, called the Series Finder, finds ordered sets of samples for the purpose of addressing biological questions pertaining to changes over a numerical property such as time. These tools were the result of a three-day-long NCBI Codeathon in March 2019 held at the University of North Carolina at Chapel Hill.


Amendments from Version 1
In this revision, we present a number of significant updates to these tools. First, in order to facilitate the use of these notebooks, we have made them available to run in the cloud via Google Colab. Second, the queries now utilize the ontology graph structure to return samples that are annotated as an ancestral term to the query term according to the ontology graphs and thus, the queries return more results. Third, in the Case-Control Finder, we have implemented the ability to match cases to controls by sex and age in addition to tissue and cell type. Fourth, we've coalesced many of the notebook cells in order to make the tools simpler and easier to use. Fifth, users can formulate queries using ontology term ID's (e.g. "DOID:3571", the term ID for "liver cancer"). Sixth and finally, we fixed Figure 2C. In the previous version of the manuscript this figure displayed data for the incorrect subset of patients.

Introduction
The Sequence Read Archive (SRA; Leinonen et al., 2011) is a large public repository that stores next-generation sequencing data from thousands of diverse scientific investigations. Despite its promise, reuse and re-analysis of SRA data has been challenged by the heterogeneity and poor quality of the metadata that describe its biological samples (Gonçalves & Musen, 2019). Recently, the MetaSRA project (Bernstein et al., 2017) standardized these metadata by annotating each sample with terms from biomedical ontologies including Cell Ontology (Bard et al., 2005), Uberon (Mungall et al., 2012), Disease Ontology (Schriml et al., 2019, Cellosaurus (Bairoch, 2018), and the Experimental Factors Ontology (Malone et al., 2010). The MetaSRA also features an interface (http://metasra. biostat.wisc.edu) for querying human RNA-seq samples using these ontology term annotations. However, the MetaSRA web interface is not capable of producing structured datasets such as those that match case samples associated with a target condition or disease with healthy control samples. Similarly, the MetaSRA is also not capable of searching for samples associated with a particular condition and/or tissue-type that are ordered according to a numeric property (e.g., age).
Construction of such datasets is non-trivial and requires further processing of the results provided by the MetaSRA website. Specifically, finding case and control samples for a given disease requires matching case samples to control samples according to their tissue or cell type. For example, if one were to naively search the MetaSRA for "liver cancer" samples, the results would include samples from Kim et al. (2020), which consist of isolated T cells from liver tumors. Therefore, only matched T cell samples would make for appropriate controls. Furthermore, given these search results, users may wish to further filter samples according to whether they are poorly annotated (i.e. , are missing cell type or tissue information), whether they are derived from a cell line, or whether they were experimentally treated. Moreover, given these results, the user may wish to explore other ontology terms associated with the search results within either the case or control samples to check for any variables that may confound downstream analyses.
Finding longitudinal or time-series data presents similar challenges. To the best of our knowledge, no existing tool addresses these tasks.
To address these two tasks, we produced two Jupyter notebook-based tools. The first tool, called the Case-Control Finder, searches the SRA via the MetaSRA terms to produce matched-case and control samples for a given disease or condition where the cases and controls are matched by tissue and cell type. The second tool, called the Series Finder, finds ordered sets of samples for the purpose of answering biological questions pertaining to changes over a numerical property (e.g., time). More specifically, the Series Finder produces ordered sets of samples, where the order is determined based on a temporal property in the metadata as standardized by the MetaSRA's real-valued properties. Examples of temporal properties include the age of a person from which a given sample originated or the time in which a given sample of cells have spent differentiating in vitro. These tools promise to facilitate the construction of suitable public datasets for secondary analyses.

Methods
The tools presented in this work were written in Python (v3.6) and make use of Python packages pandas (McKinney, 2011), Matplotlib (Hunter, 2007, and seaborn (https://seaborn.pydata. org). These notebooks can be run in the cloud via Google Colab. A link to these notebooks can be found in the README within the Github repository (https://github.com/mbernste/hypothesisdriven-SRA-queries).

Case-Control Finder
The Case-Control Finder implements the following steps to produce a dataset of matched-case control samples for a given disease ( Figure 1A): 1. Generate candidate case and control samples. Generate the set of candidate case samples by querying for all samples associated with a user-specified condition or disease using the MetaSRA-mapped ontology terms. Also, find all candidate control samples that are not associated with the target condition/disease.

Filter poorly annotated samples.
Filter samples based on a metadata completeness threshold, which requires that all samples be associated with either a tissue term or a cell type term. The tissue/cell type information is required for downstream matching of case samples to control samples.

Apply user-specified filters.
Further filter samples according to user-specified filtering parameters. The user can filter out cell line samples, treated samples, and in vitro differentiated samples. The user can also remove all diseased samples from the candidate control samples for the purpose of generating a healthy control-set.

4.
Match by tissue, cell type, age, and sex. The candidate case samples are then matched with the candidate control samples by their tissue and cell type terms. Optionally, the user can also match samples by age and sex. Specifically, given that each sample can be associated with ultiple ontology terms in the MetaSRA, a set of case samples is matched with a set of control samples when both sets of samples are labelled with the same set of tissue and cell type terms. For example, a set of case samples annotated with the set of terms "liver" and "epithelial cell" will be matched only to control samples also labeled strictly with these terms (Figure 2A). This ensures that case samples are matched with maximally similar control samples and mitigates matching samples from different tissue-types. For example, a set of case samples labelled with both the terms "liver" and "epithelial cell" will not be matched with a set of samples labelled only as "epithelial cell," as there is no guarantee that the latter set of samples originate in the liver.
Once the dataset is constructed, the notebook enables the user to explore the samples for other MetaSRA mapped ontology terms within the data ( Figure 2B and C). By presenting other common ontology terms in the data, the user may be able to identify variables that either confound analysis.

Series Finder
The Series Finder finds RNA-seq data samples that are associated with a numerical property (e.g., age or time point) for a given tissue or cell type. To do so, the Series Finder utilizes the real-value property annotations provided by the MetaSRA where each real-value property in the MetaSRA is structured as a tuple consisting of a property name (e.g., age), numerical value, and unit (e.g., year).
To perform a query, the user provides an ontology term, such as a tissue or cell type, as well as a property name and unit. The Series Finder then finds all samples that are associated with the target ontology term and real-value property. The user can also specify a set of filters (e.g. for filtering diseased samples or cell line samples) and the Series Finder will remove all samples that meet the filter specification. The Series Finder will then return all remaining samples ordered by their associated numerical values ( Figure 1B).

Results and use cases
We used the Case-Control Finder to query for samples of liver cancer RNA-seq samples matched with healthy control samples. This query resulted in 21 sets of samples representing different tissues or cell types including epithelial cells, hepatocytes, stem cells, and liver tissue (Figure 2A). The Case-Control Finder identified common terms associated with the case "liver cancer" samples ( Figure 2B), and categorized these samples by cell line status, sex, developmental stage, and treatment status ( Figure 2C). We used the Series Finder to find all brain samples in the SRA ordered by the age of the sample donor. This query resulted in samples spanning many ages ( Figure 3A). This dataset could prove useful for exploring gene expression-based signatures of aging. The Series Finder also identified common terms at each age ( Figure 3B) and for each age's sample-set, categorized those samples by cell line status, sex, developmental stage, and treatment status ( Figure 3C).

Conclusion and future work
We implemented two Jupyter notebooks for performing hypothesis-driven queries of public RNA-seq samples in the SRA. These tools are built upon the standardized metadata provided by the MetaSRA project and enable querying of the metadata beyond what is natively possible via the MetaSRA website interface. Given the SRA accessions of the RNA-seq samples that these tools produce, a user can then retrieve the gene expression data for these samples in order to perform secondary analyses. Specifically, the user can either download and process the raw reads from the SRA, or they can obtain preprocessed gene expression profiles from recent mass preprocessing efforts such as recount2 (Collado-Torres, 2017), ARCHS4 (Lachmann et al., 2018), and refine.bio (Greene et al.). Finally, these notebooks come pre-packaged with metadata files from the latest version of the SRA, as provided by the SRAdb (Yuelin et al., 2013), and MetaSRA. When the MetaSRA releases a new version of annotated metadata, these notebooks will be updated to track the new release.
We also note a few limitations to this work. First given that the MetaSRA annotates the SRA samples using an automated computational pipeline, its annotations contain some errors. Errors in the MetaSRA may propagate to the results produced by these tools, and thus, the datasets produced by these tools are best utilized as sets of candidate datasets for downstream analysis. We point the reader to Bernstein et al. (2017) for an analysis of the MetaSRA's accuracy. We also note that the SRA stores sequencing data for both bulk RNA-seq and single-cell RNA-seq samples; however, this information is not encoded in any standardized way within the SRA nor is it captured by the MetaSRA. Thus, results returned by these tools may include a mixture of both single-cell and bulk data. For these reasons, we encourage users to validate the results returned by these tools by consulting their entries in the SRA before proceeding with downstream analyses. Lastly, to facilitate access to these tools, it would benefit to implement them within an easy-to-use web interface rather than Jupyter notebooks. Future work will entail either integrating these tools into the MetaSRA website, or implementing a stand-alone web application for these tools using a platform such as R Shiny.

Data availability
The figures and datasets produced in the analyses can be found on GitHub: https://github.com/mbernste/hypothesis-driven-SRAqueries/tree/master/results

1.
While the provided examples work well, there are limitations to unfamiliar users and failures in cases that seem on reading the paper like they should work.
-For example: in series finder if I change `term` to "heart" (instead of "brain"), almost all subsequent cells fail.
-In case-control finder, if I change `condition` to "brain cancer", all but one samples returned are controls (which does not align with what is in the SRA?) and visualization formatting becomes difficult.
-By clarifying what user options are (or examples) for each place where user is free to play with the input, this could be avoided. Similarly, functions lack documentation and examples here or checks on input within the functions, so diving into the code becomes critical for use, which will limit users. Adding documentation and checks for user input could assist in this overall.

Matthew Bernstein, Morgridge Institute for Research, Madison, USA
We greatly appreciate the reviewer's valuable suggestions and feedback. Please see our responses below: 1. We agree that using the MetaSRA's API would be a great idea; however, the API restricts queries that return too many results. Specifically, for queries that return too many results, the API returns an error message that the search results are too large. This severely restricts our ability to use the API for these tools. We note that the MetaSRA is released in discrete chunks and does not track every ongoing change to the SRA; thus, whenever the MetaSRA version changes, we will update the static version of the MetaSRA packaged with these tools. We have added text to this manuscript detailing our commitment to performing these updates. Lastly, we added text to the README that makes it more explicit to the user which version of the MetaSRA these tools are utilizing. 2.
-We tested the query "heart" and it now should return results. We also provide more thorough input validation for cases in which the query does not return results.
-We have updated the code so that the tools retrieves sample that are annotated as an ancestral term to the query term (e.g. a sample labelled as "brain glioma" should be retrieved when the user inputs the query "brain cancer"). Now the query "brain cancer" will retrieve many more samples than before. We do note a few issues with the particular query "brain cancer" (which maps to term DOID:1319 in the Disease Ontology). Specifically, we found that the MetaSRA failed to label many samples as "brain cancer" due to the fact that many of the subterms (e.g. "brain glioma") are missing important synonyms that would have led the MetaSRA to pick them up. For example, the term "brain glioma" (DOID:0060108) is not associated with the simple synonym "glioma" and thus, unless a sample for a given glioma sample was described using the string "brain glioma", which appears to be rare, the MetaSRA failed to annotate this sample as a "brain glioma". Instead, the MetaSRA labels glioma samples using an alternative "glioma" term from the Experimental Factors Ontology (EFO:0005543), which does not have "brain cancer" as an ancestor term, but instead has "brain neoplasm" as an ancestor (EFO:0003833). This case points to the fact that there is still work to be done in both standardizing the metadata in the SRA and in constructing comprehensive ontologies. Unfortunately, these issues remain out of the scope for this work; however, we now include new text in the Conclusion section that discusses how the original MetaSRA annotations contain some errors and that these errors may propagate to the output of these tools.
-Thank you for this suggestion. We have added more detailed instructions for each input parameter. We also perform more thorough input-validation on the user's input. Lastly, we have added more documentation to each function in utils to help a user who wishes to dive further into the code.
Responses to minor issues: I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
in the Conclusion section and again encourage users to validate the results returned by these tools before performing downstream analyses.
8. In the Conclusion section, we now point the reader to databases of pre-processed SRA data including recount2, ARCHS4, and refine.bio. From these resources, users can download pre-processed expression data for the samples returned by the tools presented in this work.