SCUBA implements a storage format-agnostic API for single-cell data access in R

Introduction

The rapidly evolving landscape of single-cell sequencing methods has led to the production of increasingly large and diverse single-cell datasets, greatly improving our knowledge of both inter- and intra-patient heterogeneity in a wide range of diseases and normal tissues.^1,2 While there are many excellent tools available for analyzing single-cell datasets in both the Python and R ecosystems, interoperability is hindered by the use of incompatible object classes (Figure 1A). Analysis tools generally only accept one object class, forcing users to commit to a suite of packages at the beginning of an analysis. This restriction limits access to tools outside that suite, creating “walled gardens” that pose several challenges for single-cell analysis. Single-cell analysis can be inaccessible to bench scientists due to the programming experience required, and different object formats make it even more difficult for biologists to analyze data. The implementation of popular object formats in both Python and R requires users to be fluent in both programming languages, and it is difficult and time consuming to learn both without formal education in data structures and syntax in each language. The widespread use of multiple object formats also makes it difficult to validate results produced by one analysis suite with another, and visualizations produced with different analysis suites are not consistent. Additionally, the practical analysis of objects with large numbers of cells requires a way to interact with on-disk matrices rather than loading all data in memory. On-disk matrix implementations are analysis suite-specific, which introduces additional barriers to effective analysis. For example, anndata objects are natively stored in the memory-efficient HDF5 format, but anndata objects are not compatible with the Bioconductor’s single-cell tools or Seurat. If a user converts an anndata object to Seurat or SingleCellExperiment format, they must use a different on-disk matrix implementation specific to that format, which further restricts the analyses that can be performed. If single-cell object formats were interoperable, it would be easy for researchers to analyze data from any single-cell dataset, regardless of the object format used when the dataset was generated, but unfortunately this is not the case.

Figure 1. SCUBA addresses challenges posed by multiple object formats in single-cell sequencing data.

A) Raw single-cell sequencing data is stored in defined object classes, and processed downstream by packages that only accept one object class. This creates “walled garden” analysis suites of incompatible packages that complicate single-cell analysis. When a specific downstream package is desired, the user will need to convert between object formats prior to use. This is possible, but the process is difficult and error-prone. B) SCUBA returns feature expression data, metadata, and reduction coordinates from Seurat, SingleCellExperiment, and Anndata objects in a consistent output format. An overview of each object structure is shown, with rectangles indicating data matrices stored in each object. Dimensions of the matrices are labeled with “cells” or “genes” (features), and matrices placed adjacent to one another indicate requirements that matrices have the same number of values in the dimension indicated (i.e. for Seurat objects, the “reduction coordinates” and “gene expression” matrices must have the same number of cells, but may have varying number of genes (or in the case of reduction coordinates, dimensions). Next to the description of each matrix, object-specific code to retrieve the matrix is given. If additional modalities are supported by an object type, the structure of matrices specific to modalities are shown, along with code to retrieve data on alternate modalities. The output format for SCUBA is shown at the bottom of the panel. The output is a single R data.frame with values for each variable requested for each cell. The S3 methods added by SCUBA to yield the output format are shown in blue, and are based on the existing FetchData generic and method from Seurat.

Currently, the most effective solution is to manually convert between object classes. All major single-cell analysis packages implement conversion functions, and third-party packages such as sceasy³ are specifically designed for these conversions. However, inconsistencies in approaches to object structure across implementations often result in data loss upon conversion, which is difficult to overcome. Additionally, the rapid development of these packages means that conversion functions are difficult to maintain and frequently break due to changes in other packages. This is especially true when converting to and from the anndata format, since this format is implemented in the Python programming language, and the Seurat and SingleCellExperiment objects are implemented in the R programming language. Even if conversion is successfully achieved without loss of data quality, it has recently been demonstrated that results from Seurat differ from those of Scanpy,⁴ despite the fact the two packages implement ostensibly identical processing steps. Addressing interoperability and consistency issues between analysis suites is crucial to ensuring the fidelity and reproducibility of single-cell analysis results, making consistent visualizations across suites essential.

Rather than conversion between objects, we propose a more sustainable approach to the visualization and further analysis of pre-processed single-cell data by implementing a unified API for all common single-cell object formats. Here, we present Single-Cell Unified Backend API (SCUBA), an R package based on the data accession function in the widely used Seurat⁵ package that returns data from Seurat, SingleCellExperiment, and annadata objects in a common format for downstream visualization and analysis (Figure 1B). SCUBA also implements new data-specific access functions for all supported object types. Data is returned in a single R data.frame, with requested variables as columns, and cells as rows. The functions in this package allow users to plot data in a consistent manner from these object types in R, without requiring conversion. SCUBA can also be used in functional programming applications as the basis for single-cell plotting packages, or in the development of Shiny apps. For objects with very large numbers of cells, it is now possible to choose the object class based on on-disk storage performance and produce visually consistent plots without having to downsample the object. Packages and scripts created with SCUBA are flexible with regard to input type, greatly improving the consistency of results between objects and increasing accessibility of these analyses for non-programmers.

Methods

Use cases

The features of SCUBA fall broadly into three categories; data access, data visualization, and data exploration. The functions provided in these categories can be used independently or in a stepwise pipeline. Generally speaking, SCUBA works best for objects that have been filtered and clustered, though SCUBA can work on objects in any state as long as the data being requested exists. Here we highlight independent use cases using a downsampled version of the acute myeloid leukemia reference dataset¹⁰ generated by Triana et al.¹¹ Additional vignettes are provided on the SCUBA GitHub page.

FetchData Methods for SingleCellExperiment and Anndata Objects

Example usage of SCUBA’s FetchData methods is given Figure 2. The existing Seurat method (first column) is compared to the methods added by SCUBA. There are only minor variations in input syntax across the three supported object types, and the required parameters are few, making the methods easy to use. All data requested is specified using the vars parameter. The methods infer whether the data requested is metadata, reduction coordinates, or feature expression by parsing the character vector passed to this parameter. To retrieve feature expression or reduction coordinates, the user adds a “key” with an underscore giving the name of the reduction, or the modality to pull feature expression data from. If using an object with only one modality, the modality key is not needed, and the key is also not needed to retrieve metadata. Minor variations in the key exist between object types, due to object-specific conventions for naming modalities (which are called “assays” in Seurat objects, and “Experiments” in SingleCellExperiment objects). Variations in the layer parameter are based on variations in conventions for naming layers (in SingleCellExperiment objects, “assays”, and in Seurat v4 and earlier, “slots”). The consistency in parameters between the three object types, and the presence of only minor differences in inputs to each parameter, facilitates the writing of scripts for any object type.

Figure 2. The methods added by SCUBA simplify the retrieval of data from supported object classes.

The existing seurat method (first column), is compared to the methods added by SCUBA for SingleCellExperiment and anndata objects (second and third columns). The methods use consistent syntax across object classes, and involve the use of only a few parameters. Pseudocode is used in the examples. object represents a single-cell object. features represents one or more features, from any modality in the object. metadata represents one or more metadata variables, for example, cell type classifications. reduction_dims represents a set of dimensions in a reduction included with the object, with the number of the dimension separated from the reduction with an underscore. For example, to fetch the first and second dimensions of the UMAP projection, reduction_dims would be c(“UMAP_1”, “UMAP_2”).

The output of FetchData is identical across the three object classes. The output is an R data.frame with values for each requested feature in vars per cell. Columns represent each feature, and rows represent cells.

Metadata, reduction-specific accession methods

SCUBA also includes S3 generics and methods specific to the retrieval of metadata and reduction coordinates, which are faster than retrieving the same data via FetchData for R object types. An overview of the fetch_metadata and fetch_reduction functions is given in Figure 3A. As with FetchData, the output of fetch_metadata and fetch_reduction is an R data.frame with data for the requested metadata variables or reduction coordinates, respectively, as columns, and rows for each cell. Figure 3B compares the usage of fetch_reduction and fetch_metadata between supported object types. We implement these methods in anndata objects for consistency in syntax, but their performance is roughly equivalent to FetchData. The functions are easy to use, and the inputs to each function do not vary based on object type. To set defaults for the reduction and cells parameters of fetch_reduction, SCUBA provides several utility methods. default_reduction will search for UMAP, t-SNE, and PCA reductions, and will return them in that order if they exist. get_all_cells will return the IDs of all cells in the object.

Figure 3. SCUBA methods specific to the retrieval of metadata and reduction coordinates.

A) Overview of outputs of fetch_metadata, for metadata variables, and fetch_reduction, for reduction coordinates. The output is an R data.frame with the metadata or reduction coordinates as columns, and the cells as rows. B) Comparison of the usage of fetch_metadata and fetch_reduction across each object type. For fetch_metadata, the metadata variable or variables to retrieve (which are represented as metadata in this pseudocode example) are specified via a character vector input to vars. For fetch_reduction, the dimensions to return from the reduction coordinate matrix is passed to dims, and the reduction to pull from is specified via reduction. The cells parameter allows the user to specify which cells to fetch reduction coordinates for. For ease of use and flexibility, there is no difference in inputs between object types; only the object itself varies.

Figure 4A-B compares the performance of fetch_metadata and fetch_reduction with the performance of FetchData to pull one metadata variable, and the first and second dimensions of UMAP coordinates, respectively. Run time was tested for each function on random subsets of varying numbers of cells, with five subsets created for each size. The fetch_metadata and fetch_reduction methods were more performant than FetchData in Seurat and SingleCellExperiment objects for all subsets tested. In anndata objects, the runtime of these functions was comparable to that of FetchData. Performance testing for the FetchData methods added by SCUBA was also performed (Figure 4C). Performance of the method for anndata objects exceeds the performance for the existing Seurat method in most cases, and the performance of the SingleCellExperiement method exceeds performance of the Seurat method for the largest subset tested (500k cells).

Figure 4. Performance testing of SCUBA functions and methods.

Five random subsets of the indicated numbers of cells were created from the Human Brain Atlas object downloaded from CellXGene.²¹ The subsets were saved in the following object formats: Seurat, via saveRDS(), SingleCellExperiment, via HDF5Array::saveHDF5SummarizedExperiment(), and anndata, via write_h5ad. For all tests, the indicated operations were run on each of the five subsets, and the run time was measured using the tictoc²² package. A) Comparison of FetchData methods vs. fetch_metadata for the retrieval of data for a single metadata variable. In most cases, using fetch_metadata to pull metadata was more performant than using FetchData. B) Comparison of FetchData methods vs. fetch_reduction for the retrieval of data for a pair of reduction coordinates. fetch_reduction was more performant than FetchData for the retrieval of reduction coordinates in most cases. C) Performance of the FetchData methods developed for SingleCellExperiment and anndata objects, compared to the existing FetchData method for Seurat objects. A single feature was pulled via FetchData for each of the random subsets for the indicated object and number of cells.

Example scripts created with SCUBA

Figure 5 gives an example usage of SCUBA methods to create plots with consistent visuals across object types. Figure 5A shows the scripts to create a density plot showing expression by cell type from each of the three supported object types, showing regions of the script that vary between object types, and regions that are conserved. Figure 5B shows the output of the example script. The script demonstrates the ease at which expression data can be visualized from each object format, and the ease at which plot visuals can be harmonized across object formats.

Figure 5. SCUBA enables flexible plotting scripts harmonized across object types.

A) Example script for visualizing expression of a gene by cluster in a density plot for each of the three supported object types. The three boxes for FetchData indicate slight variations in the script for each object type. All downstream code is the same across object formats. B) Output of the plotting script in (A). Output does not vary by object type.

Any plot visualizing a combination expression data, metadata, and reduction coordinates can be created by generating a table from FetchData, fetch_metadata, or fetch_reduction, and passing the output table to downstream plotting code. Plotting is performed via ggplot2¹⁷ in this example, but any other plotting package that accepts a data.frame or a tibble as input may be used. If desired, it is also possible to convert to a pandas¹² dataframe via Reticulate,⁷ and perform plotting operations in python. The flexibility of SCUBA’s data access methods facilitate the creation of a broad variety of plots from single-cell data.

Figure 6 shows an example script that simplifies the printing of unique values of a metadata variable represented in an object, which is a commonly used basic operation in analysis. With SCUBA, this operation can simply be performed by calling fetch_metadata on the object and piping the results to unique(). The language used is the same for all supported object classes, which negates the need to memorize and use the most efficient function calls for each respective object type.

Figure 6. SCUBA simplifies common object exploration operations.

This figure compares the usage of SCUBA with the most efficient equivalent operations for viewing the unique values of a metadata variable represented in an object. The operation with SCUBA is shown in the first column, and the most efficient equivalents are shown in the second column. SCUBA simplifies this operation, allowing for the development of scripts that are generalized for multiple object types.

Conclusions

SCUBA addresses issues with interoperability between single-cell object formats by providing a flexible backend that returns data in a consistent format, via a consistent interface. The consistent output format of SCUBA facilitates downstream use in functional programming applications (plotting scripts, packages, etc.) and allows for consistent visualizations across object types. Packages and scripts using SCUBA will not require object conversion prior to use, conferring several advantages for end users. Users will not have to risk data loss upon object conversion, and analysis will be more straightforward without conversion, requiring less programming experience. Packages made with SCUBA will also allow users to choose object classes based on storage and performance characteristics that are best for the specific dataset, rather than being constrained to a class based on downstream packages. SCUBA does not allow users to use any analysis package with any object format, however. The aforementioned benefits only apply to packages and scripts created using SCUBA. SCUBA also only performs data access operations, and is not for object assembly, clustering, or filtering. Because of this, SCUBA is not a replacement for analysis packages such as Seurat and Scanpy. Instead, SCUBA allows users to visualize objects that have been prepared with these analysis packages in the same manner, regardless of object class.

Support for MuData^18,19 will be added in the future, as this Python object class is especially useful for storing data from multimodal single-cell sequencing experiments. SCUBA is particularly well suited for interactive use, such as in Shiny apps, where multiple object formats may be used as inputs. We developed a single-cell browser, scExploreR,²⁰ that allows users to create consistent Seurat-style visualizations from either Seurat, SingleCellExperiment, or anndata objects. SCUBA can also be used to create a plotting package that produces visuals from any supported object class for reports and shiny apps, and a QC package reporting the results of preprocessing steps such as filtering, clustering, and batch correction could also be created using SCUBA. The flexibility of SCUBA is envisioned to facilitate analysis and visualization of preprocessed data, unifying disparate object-based package ecosystems.

Ethics and consent

Ethical approval and consent were not required.