Community-driven ELIXIR activities in single-cell omics

Technologies

Single-cell omics technologies have seen widespread adoption since its announcement as Nature Method of the Year in 2013.¹³ The most widely used SCO technologies are single-cell RNA-seq, single-nucleus RNA-seq, and single-cell ATAC-seq. The early days of these technologies were dominated by heterogeneous implementations of handling, preparation and sequencing protocols,¹⁴ which left its mark in the large number of software tools that had been developed, which in part led to lack of standardization of data and metadata. In recent years, we have seen a number of technology providers prevail for instrumentation (e.g. Fluidigm, 10x Genomics), reagents (e.g. ThermoFisher, QiaGen, Roche), and sequencing (e.g. Illumina, MGI, ONT and PacBio). This has led to the community converging around a few workflows based around the popular 10x Genomics Chromium and SmartSeq chemistries (Figure 1c), which have been exploited for large scale sequencing efforts such as the Human Cell Atlas (HCA)¹⁵ and Human BioMolecular Atlas (HuBMAP),¹⁶ that, in turn, have resulted in large investments into solving the sample handling, data integration and data management problems underpinning the vast array of data being generated.

While it has been a decade since the SCO technologies have taken the center stage in unraveling bio-molecular heterogeneity, technology developers are far from stagnant and we are seeing rapid evolution of these technologies. We witness the adoption of single-cell genomics (e.g. MissionBio Tapestri) and some progress in the field of single-cell proteomics.¹⁷ Single-cell multimodal omics was announced as Nature Method of the Year 2019.¹⁰ These assays provide multiple readouts that can be used to define cells. For example, antibody profiling (e.g. CITE-seq¹⁸) allows scientists to contextualize novel cell types and states in the context of well-established cell biology markers; immune repertoire profiling can also link how transcriptional profiles of immune cells differ based on the receptor specificity¹⁹; true multimodal omics such as simultaneous profiling of chromatin accessibility, DNA methylation and transcriptomics (e.g. scNMT-seq²⁰) allows us to also decipher the regulatory changes that underpin the transcriptional landscape of cells.

In 2020, spatially resolved transcriptomics (SRT) was announced as Nature Method of the Year.²¹ These technologies fall under three broad areas of laser capture microscopy combined with single-cell sequencing (e.g. GeoMX DSP, Tomo-seq), in situ capture arrays (e.g. ST, Visium, Slide-seq, HDST), and image-based single-molecule expression quantifications (e.g. in situ sequencing,⁸ seqFISH+,²² Molecular Cartography,²³ MERFISH²⁴). Despite its youth, there are over 20 SRT profiling technologies,²⁵ which are also expanding into other omics modalities, such as proteomics (e.g. CODEX²⁶) and metabolomics.²⁷ However, the community quickly realized the potential of this technology and early efforts were pushed by the Chan Zuckerberg Initiative to harmonize the field by funding efforts such as StarFish,²⁸ a platform to uniformly process raw single molecule SRT data, and the SpaceTX consortium, which aim to benchmark and harmonize data from various SRT platforms and analytics methods. Efforts to make comprehensive cell atlases available to the community now facilitate more single-cell study designs to include perturbation and lineage tracing experiments (e.g. via CRISPR^29–33).

While the standards for scRNA-seq have by-and-large converged, the extension of single-cell technologies to new modalities and experimental setups places even more emphasis on establishing adaptable and extensible standards.

Analysis tools

There is a large number of single-cell analysis methods and tools available that cover a wide range of analysis steps. In January 2022, the scRNA-tools database recorded nearly 1,200 tools divided over more than 30 categories (Figure 1b and d).³⁴ Computational and analytical challenges in single-cell genomics have been discussed extensively.^11,35 The analysis steps vary depending on the modality of single-cell data. For the most widely used modality, transcriptomic data, the community has converged on a consensus regarding the analysis steps.³⁶ Yet, even some foundational steps remain active areas for research, such as how to normalize scRNA-seq data^37–39 or how best to perform differential expression analysis.^40,41 Also, annotation of cell types varies drastically between studies, with many resorting to ad hoc decisions. A complete atlas of all cell types would be required to improve the standardization of cell-type nomenclature and ontologies (e.g. Cell Ontology, UBERON).⁴² Scientists analyzing SCO data need to have sufficient information on the strengths and limitations of the available analysis tools in order to select the most suitable ones for their data and purpose. However, systematic comparison of these tools is challenging, especially given the ever-increasing number of methods and their parameter combinations.

Studies have identified essential guidelines for benchmarking computational methods^43–49 and reviewed published benchmarking studies of computational tools for omics data, highlighting the advantages and limitations of benchmarking across various domains of the life sciences.⁵⁰ Systematic benchmarking frameworks can enable crowdsourcing and community challenges, which have been a successful means for fostering community creativity and expertise to address open problems.⁵¹ In a concerted effort to address “grand challenges” for the SCO community,¹¹ the Open Problem in Single-Cell Analysis group⁵² is devising competitions to address those challenges, e.g. the Multimodal Single-Cell Data Integration competition (NeurIPS 2021).

Despite community efforts, the major challenges facing SCO benchmarking studies are the lack of appropriate experimental data and/or realistic simulated data that can be used for benchmarking, as well as the lack of agreed-upon measures to evaluate different methods. There is also a need for a common platform to conduct benchmark studies. The Open Problems NeurIPS challenge provides a leading example for evaluating methods using common datasets, performance metrics, as well as providing a compute infrastructure to run these methods. However, there is still a need for platforms that allow for continuous update of results as new tools and/or metrics become available and to dynamically respond to the needs of individual communities within the life sciences.

Currently available tools and pipelines differ in their usability. While the majority require programming knowledge, several pipelines provide GUIs for users without programming experience (e.g. Galaxy, Chipster).^53,54 Most tools are available as R and Python packages or as a collection of scripts on GitHub (Figure 1e). To keep up with the technology developments, these methods and tools are continuously updated. Yet, maintaining tools and providing support is often challenging for research groups. Interoperability between methods and tools is limited despite efforts by popular packages such as Seurat⁵⁵ to provide wrappers around other tools. However, frequent updates to tools to keep up with technology developments (e.g. updating single-cell objects to cater for multi-modal data) limits interoperability, emphasizing the importance of a concerted effort to address robust data and metadata standards.

Standards and research data management

The major factor in realizing interoperability is the definition and adoption of robust data format standards. While more than 1,000 SCO tools exist, there is broad acceptance of widely adopted raw data standards (e.g. FASTQ, FAST5, BAM, CRAM) and convergence to a few processed data formats (e.g. tab-separated files, AnnData, HDF5, loom, SingleCellExperiment, Seurat). The data formats and structures employed by some of the most popular tools for SCO data analysis^55,56 have had to change to adapt to new technologies that rendered previous formats inadequate. While these changes in data formats are frustrating for maintaining data analysis workflows, they are necessary for keeping up to date with the rapid technological developments in this field. This places a strong emphasis on planning to adapt to changes by employing extensible structures that do not break the chain of backwards compatibility.

Furthermore, metadata standards and minimal reporting guidelines enable the appropriate archiving and subsequent reuse of SCO data. For some specific library construction or sequencing technologies, provision of platform specific metadata is routine and standardized (e.g. for the 10x Chromium), however additional care is required for in-house solutions and for reporting metadata for other parts of the experimental design. Establishing the Minimum INformation about a SEQuencing Experiment (MINSEQE) guideline was an important achievement for reporting metadata for sequencing data.⁵⁷ Recently the Minimum Information about a Single-Cell Experiment (minSCe) guidelines were established,⁵⁸ which defines 48 attributes that describe the biosource, isolation method, protocols, library construction, sequencing assay, raw data files and sequences, and cell- and sample-associated information derived from data analysis. However, as we move towards atlasing entire organisms and the rapid emergence of spatially resolved SCO technologies, further refinement of these minimal reporting standards is required to allow for describing common landmarks to facilitate integration of reference maps at differing scales into a single common framework, e.g. through the adoption of the Common Coordinate Framework⁵⁹ that aims to uniquely and reproducibly define any location in the human body. A major consequence of establishing robust metadata standards is that they facilitate the establishment of SCO portals that provide access to uniformly processed data from a wide variety of SCO studies (e.g. the EBI Single Cell Expression Atlas, the Broad Single Cell Portal, and the HCA Data Portal). Such portals rely on accurate and sufficient metadata to enable appropriate processing of SCO data from a wide variety of studies in a uniform way.

Given the fast pace of technological developments in the SCO field, the community has identified that both adaptability and extensibility are key considerations in defining sustainable standards. This has been achieved in the field of medical imaging with the Digital Imaging and Communications in Medicine (DICOM) format,⁶⁰ which has been constantly extended and updated without breaking backwards compatibility for nearly 30 years. However, this level of flexibility was only achieved by the third version of the DICOM standard, 10 years after its initial inception, and it was a concerted effort between medical and trade associations. Part of the successful adoption of the DICOM format is that despite all major medical imaging players having their own proprietary formats, they provide an interface to the DICOM format. In order for the SCO community to reach a similar level of interoperability as has been achieved in medical imaging, technology providers and tool developers should also either adopt the most common standards in the SCO community, or provide interfaces to them. While it is not clear how the current landscape of SCO data and metadata standards will stand the test of time, some aspects that will determine their success with be their ability to adapt to change (e.g. through using extensible formats such a JSON), used of controlled nomenclature (e.g. utilising ontologies for defining attributes), and adopting versioning (e.g. semantic versioning).

Training

Upskilling life scientists to analyze SCO data is a moving target, given the fast development of the field. The cutting edge analysis methods for SCO data tend to be rather computationally complex, making them harder to grasp for life scientists who typically lack a solid background in mathematics, statistics and machine learning and often R/Python skills too.

Trainers, on the other hand, find themselves updating training materials constantly and, in general, struggle to keep up with the fast development of new analysis methods in order to choose what to teach. To make things worse, often only a small fraction of their working time is dedicated to training, or training is offered on a voluntary basis on top of their workload. It is therefore not surprising that even though single-cell courses are offered by several ELIXIR Nodes, many flavors of SCO are not yet covered. For example, courses on single-cell epigenetic, multi-omics as well as image-based spatially resolved SCOs are still rare. The demand for training continues to grow, but the lack of competent experts with enough training experience and time available is a major bottleneck in scaling up training provision. While pedagogical train-the-trainer (TtT) courses^61,62 can empower experts to feel more comfortable to teach, the constant evolution of the SCO field can intimidate newcomers.

There are also more practical hurdles: the analysis of single-cell data requires a sophisticated computational environment with many tools and their dependencies, often requiring high-end computational resources. These environments have to be ready-to-go or at least easy to set up, and reproducible across heterogeneous hardware infrastructure, allowing the participants to re-run the practical and to analyze their (probably much larger) own data in their own setting. It is also challenging to find good training datasets that are small enough to be run in a class but meaningful enough to prove the concepts.

Training platform

Upskilling scientists in SCO data analysis and standards lies at the heart of the ELIXIR SCO Community, and particular efforts will be made to make the training scalable and FAIR in coordination with the ELIXIR Training platform (Table 2).

The SCO Community will ensure that training materials and expertise are shared efficiently and following FAIR and open research principles.^63,64 We will collaborate with ELIXIR’s Training Portal TeSS⁶⁵ to establish a well-curated SCO training portal, listing national and international training providers, web resources and upcoming training events. To help the current trainers and encourage new ones, we will annotate training materials with appropriate metadata, curate training datasets, provide detailed explanation on how to run courses, and share best practices and best ways to teach the more advanced concepts. In order to identify SCO areas which lack sufficient training, we will participate in designing the annual training gap survey by the Training Platform, and also perform more detailed SCO training surveys if needed. We will regularly host trainer workshops targeting the areas identified as lacking sufficient training to exchange experiences and discuss materials.

Anyone should be able to learn about SCO data analysis independently of time and place. To make the training scalable, lectures and video tutorials will be recorded for asynchronous learning, and combined into modular eLearning courses. Resources will be gathered and annotated on a single site for easy discovery. In addition to organizing training in SCO data analysis and standards to complement ELIXIR Nodes activities, we will provide training in best practices for trainers (TtT) to increase the number of expert trainers.

The course software installation challenge will be addressed together with the ELIXIR Tools Platform, as described below, using Conda environments, containers and Notebooks. Both Galaxy⁶⁶ and Chipster⁵⁴ offer specific training access and a comprehensive collection of training materials. There is no setup required, and the same environment is available when analyzing one’s own data after the course.

Tools platform

SCO data analysis typically requires a large number of tools and their dependencies. The installation challenge can be eased by providing Conda environments⁶⁷ and containers⁶⁸ for SCO, in alignment with the work developed in the Tools Platform’s Packaging, containerisation and deployment activity. Also, RStudio or Jupyter Lab based SCO Notebooks can be made to support courses and self-study. The Community will develop cloud-deployable analysis pipelines for SCO data and make them available also through the web-based Galaxy Single Cell Omics, Galaxy Human Cell Atlas project, and Chipster analysis platforms for researchers lacking programming skills. The analysis pipelines will be deposited in WorkflowHub⁶⁹ for easy discovery, re-use and assessment.

The SCO Community will take several actions to address the aforementioned challenges in benchmarking. Liaising with data analysis experts, we will carefully curate data collections suitable for addressing specific tasks within the SCO data analysis workflow (e.g. multi-modal data integration, deconvolution of bulk data). For this, we will survey the landscape of existing benchmarking studies and identify the datasets they used and how they were evaluated. Whenever possible, our focus will be on real datasets rather than simulated ones, given the bias introduced by simulated data towards methods using the same underlying model. In order to address the lack of agreed-upon performance metrics to evaluate different types of methods, we will collect and curate existing metrics, and develop/suggest new measures when necessary (Table 2).

Regarding the need for a common platform to conduct benchmark studies, we will explore using OpenEBench.⁷⁰ This ELIXIR benchmarking platform offers a flexible computational framework that allows individual communities to design and perform their benchmarking experiments. Communities are responsible for defining the reference datasets and the evaluation metrics and designing and developing evaluation workflows. Software developers are then able to use these workflows to evaluate their tools against the reference datasets, and the computed metrics are compiled, analyzed and publicly exposed in tables and visualizations. The results of the evaluation are then used by the community or any other OpenEBench user to decide which is the most suitable tool to do their analysis. The SCO Community will provide guidelines for the setup of single-cell benchmarking experiments. The guidelines have to cover three topics: 1) the scope of the benchmark, 2) the evaluation metrics that will be used to measure the performance of the tools and 3) the reference or gold standard datasets. The SCO Community will establish a benchmarking environment for SCO data analysis tools within the OpenEBench infrastructure, to facilitate a variety of community-driven challenges to address the diversity of the SCO applications.

The SCO community will perform periodic reviews of highly performant and rapidly adopted methods for registration in the bio.tools catalogue. To this end we will work closely also with the EDAM ontology to define single-cell specific keywords, which will help us not only to annotate the tools but also tag courses in TeSS.

Compute platform

The computing resource requirements of SCO data analysis increase constantly as the scale and modality of the data sets grow. The discussion between the SCO Community and the ELIXIR Compute Platform is therefore vital to ensure sufficient resources. The Community will also benefit from the Compute Platform’s Container Orchestration task, which will allow execution of containerised software tools and workflow workloads supporting public and sensitive data across ELIXIR Nodes. The ELIXIR Authentication and Authorisation Infrastructure (AAI) will be supported in the context of sensitive SCO data and whenever controlled access will be needed, we count on learning from the HCA’s experience on this matter.

Interoperability platform

The ELIXIR SCO Community will promote the development and usage of standards of metadata and file formats to ensure reproducibility of analyses and data reuse across biological and bioinformatics research communities. We will support existing efforts for aggregating and disseminating related metadata standards, e.g. from ArrayExpress, the HCA and further efforts. This is particularly important for emerging spatially resolved data, for which we aim to investigate efficient and scalable reporting structures, in line with current efforts in imaging and omics databases (Table 2).

Data platform

An important consideration for sensitive human data is the General Data Protection Regulation (GDPR). To comply with the GDPR, raw human sequencing data deposited in EGA is protected and requires approval of the Data Access Committee Officer (DACO) as well as Data Transfer Agreements (DTA) outlining the conditions for allowing access to sensitive data. However, there is heterogeneous interpretation of the GDPR across Europe, and to facilitate this there have been a number of nationally Federated EGAs being established. Other non-human raw sequencing data would be deposited in ENA and would not be subject to these restrictions.

The ELIXIR SCO Community will encourage all data generators to ensure their data is available from an ELIXIR core data resource, or is deposited with a suitable ELIXIR core deposition resource, wherever possible, to ensure maximum data reuse and long term sustainability of all SCO data across the broader community. Via connections to key ELIXIR resources at EMBL-EBI (ENA, EGA, ArrayExpress and BioSamples database), we will promote discussions with these data resources to encourage the adoption and development of standards, where needed, to support the rapid pace of technology change in the single-cell field. We will leverage EMBL-EBI connections to the HCA Data Coordination Platform to broker the HCA data to ELIXIR core data resources and deposition resources.

Data and Software Availability

Data on scientific publications on SCO and number of cells sequenced was obtained from the Single-cell studies database.⁷⁴ Data on SCO tools was taken from the publicly available repository of the scRNA-tools database.³⁴