Ten recommendations for organising bioimaging data for archival

Introduction

Scientific data archival has a long history of providing publicly accessible storage of experimental data that typically involves manual and automated curation and annotation with appropriate metadata for reuse by others (Whitlock et al. 2010; Rausher et al. 2010; Berman et al. 2014). In the area of bioimaging, global and public resources such as EMPIAR (Iudin et al., 2016, 2023) and the BioImage Archive (Ellenberg et al., 2018; Hartley et al., 2022) provide a valuable service to the life-science community by supporting the archival and reuse of imaging data, often acquired at considerable cost, in line with the aspirations of the FAIR Guiding Principles (Wilkinson et al., 2016). There are numerous advantages and benefits to reusing bioimaging data, including more economical use of limited resources such as instrumentation and highly skilled technical staff. Moreover, specimens may be unique, costly to acquire, or difficult to reproduce, meaning that such data may only be accessible via archives. Archived data can be mined for reanalysis, verification and validation, and for development of new analytical techniques and software tools, such as machine learning model training. Reuse of such data may also lead to improvements in how it is produced, both technologically and methodologically. As practitioners in bioimaging data archiving, it is our experience that handling large datasets presents several data-management challenges, particularly in recent years with the rapidly increasing volumes of bioimaging data (Ellenberg et al., 2018). For example, it took eight years for EMPIAR to archive a total of one petabyte of data, but the second petabyte took only 14 months (Iudin et al., 2023). Bioimaging datasets may comprise numerous and sometimes very large files in a variety of, sometimes proprietary, formats. Individual files may include multiple channels and time points and data and metadata from several specimens. Besides the raw image data, there may also be a need to archive processed data, reconstructed 3D volumes, segmentations, particle stacks and other derived or related data.

There are two related but distinct avenues for organising data: labelling (metadata) and arranging data items (order). Metadata are essential to make the data useful even though metadata standards are difficult to enforce. Therefore, metadata standardisation has received a lot of attention with initiatives such as Bioschemas, an effort to improve findability of datasets via standardised textual annotations, MIAME (Brazma et al., 2001), recommendations for minimal metadata describing a microarray experiment, and the overarching FAIR Guiding Principles (Wilkinson et al., 2016). For bioimaging, REMBI (Sarkans et al., 2021) provides community-supported recommendations on how to describe all aspects of bioimaging experiments including sample preparation, data processing and analysis. Whereas there are several ongoing efforts towards standardising bioimaging data formats (OME-NGFF (Moore et al., 2021), DVID (Katz & Plaza, 2019), BigDataViewer (Pietzsch et al., 2015), etc.), we know of no efforts towards harmonising how to organise datasets for maximum usefulness for archival in mind. The organisation (order) of data is usually taken for granted and it falls upon refinements of the metadata to bear the burden of meaningfully describing the data. Nevertheless, it is essential to maintain coherence between metadata and order of the data, for example in the naming of entities to facilitate meaningful navigation between the two.

Motivation

Good organisation (order) of data improves its usefulness and is the responsibility of the data depositors. Depositors are best placed to present data in a way that adequately captures the experimental design and outcomes. For this reason, the recommendations that are outlined below are primarily targeted at data depositors as they are best positioned to structure the data meaningfully. Additionally, these recommendations are also aimed at developers of software tools that either produce the raw data (acquisition software on microscopes), process and transform it into a useful form, or extract meaningful domain insights (bioimage analysis). Organising a dataset to minimally convey a structure in line with the actual experimental output can improve its usability while the bulk of meaningful attributes can be expressed in the metadata. The degree of usefulness depends directly on the quality of organisation, and thoughtful consideration of the needs of users (i.e., those who consume the archived data) improves that usefulness. Good organisation also gives a dataset transparency and understandability: users can immediately distinguish the various experimental categories as well as plan how to analyse the data (Petek et al., 2022). Therefore, it helps to have a clear perspective of the various types of users.

In general, we consider three types of users: intra-domain scientists, inter-domain scientists and extra-domain scientists (Datta et al., 2021). (For the purpose of this article, we will refer to any such user of a dataset as a ‘scientist’, interested in extracting some knowledge from the archived data.) Intra-domain scientists are familiar with key attributes of the data and may be able to quickly assess the usefulness of a dataset. An example would be a structural biologist mining an electron cryo-tomogram to extract sub-volumes that have not been previously studied. Inter-domain scientists may want to mine the data for purposes relevant to some other domain. For instance, on the genomics side, using spatial transcriptomics imaging data for fine-grained localization of individual transcripts would be a possible scenario for an inter-domain scientist. Extra-domain scientists are only interested in data for its technical properties, i.e., for some purpose completely unrelated to the original purpose of the data’s collection. A computer scientist, for example, may want to assess the performance of a learning algorithm on fluorescent microscopy images when performing some classification task. It is likely to be a challenge to optimise the organisation of data for all types of users simultaneously. In practice, organising the data to be useful to scientists with the least familiarity with the domain will most likely advance its usefulness for all types of scientists and can thus be a good aspiration.

Organising data results in a hierarchical arrangement of data into files and folders. The visual properties of such an organisation influence the usability of the data. There are several considerations that affect the organisation:

We can refer to the above as organisational resources, and it is through their judicious use that the data can become usable. Very long file names, potentially problematic characters, and deep nesting of folders are examples of how injudicious use of these resources can result in unusable data.

A simple example of how this is useful is the way most operating systems apply ordering of a directory’s contents either lexicographically or by other attributes such as date-time stamps. These take advantage of the familiarity that users have with the conventional ordering of these attributes. In non-trivial organisational tasks, we may need to express complex relationships between the entities at hand. For instance, a dataset that consists of the experimental measurements resulting from a sequence of treatments applied on a set of specimens measured at various points in time requires the use of specimen, treatment and time-point identifiers as well as other experimental attributes (data formats, alternative perspectives, transformations of the data such as changes in units, etc.) to be captured in such a way as to preserve the main experimental relationships. In that case, we can expand our set of organisational resources to include file formats in addition to the set of symbols (letters, numerals, punctuation, uppercase and lowercase) used to create the various identifiers. Ideally, we would like to keep repetition to a minimum so that the nature of the experiment can be readily discerned.

The way organisational resources are used affects the usability of the resulting organisation: using too few of them will obscure the meaning of the organisation while using too many will overwhelm potential users. For example, including redundant folders along any part of the hierarchy (folders that contain only a single folder which in turn contains the actual data) makes it tedious to navigate through a dataset. On the other hand, dumping all files into one folder will make it difficult for the end user to distinguish between groups of semantically related files, especially when thousands of files are present. Similarly, naming files and folders by referring to entities inaccessible to their intended users (e.g., local machine names or private accession codes that external users will not have access to or even fathom) consumes precious ‘name space’ without conveying any useful information. Organising data is thus an investment of time and effort with the aim of improving the usefulness of the data.

We can therefore formulate the organisation task as follows: given a set of related data items associated with an experiment, how may they be organised to best convey their relationships using as few organisational resources as possible while maximising their usability?

To achieve this, we define the term facet to refer to the various attributes germane to the experiment which may be included in the folder and file names. A non-exhaustive list of facets is: specimen names (organism, tissue, cell type/line), experimental roles (treatments vs. controls), time (developmental status, date, elapsed time), processing status (raw data, by algorithm, procedure), commonly available experimental equipment (microscopes, detectors, preparation equipment model names), replicates, file types (3D volumes, particle stacks), names of software used for processing.

This guide attempts to solve the organisation task by providing 10 recommendations that arise from our experience of handling hundreds of large image datasets in the public archives EMPIAR, BioImage Archive and BioStudies (Sarkans et al., 2018). Ideally, we would like to organise potentially numerous and voluminous data to maximise ease of use and hence facilitate the user’s ability to:

This guide does not offer any recommendations for a detailed schema to describe experimental and analytical procedures; those may be captured in metadata for the various archives. Neither does it describe how to decide which experimental facets are appropriate (these are part of the experimental design), nor does it attempt to describe how to achieve organisation for automated analysis (we assume that the resulting organisation will be consumed by humans). It also ignores the universe of image formats in use and mainly includes examples from our experience archiving bioimaging data, but we anticipate it may be useful across other imaging disciplines. Good organisation improves data structure and format predictability and may facilitate automated processing. Therefore, our guide is intended to lead towards best practices rather than serve as a framework. Finally, this guide does not aim to achieve standardisation. We believe it is more practical to have a set of best practices and leave it up to the data authors to decide how best to apply them.

We believe that the recommendations outlined here may be of value to two principal groups of users: 1) data depositors, who need to design and prepare their data to improve its usability to the community, and 2) technologists (hardware, software and methods developers), who, by considering these recommendations in their designs, can greatly facilitate good data organisation at the source.

Recommendations

We will motivate our guide by referring to a fictitious EMPIAR dataset. This dataset has a clear structure, but we propose that it can be further improved following the recommendations in the guide below.

Our goal is to improve the file/folder structure shown in Figure 1 to better convey the relationships between the experimental facets while economising on the organisational resources available. For clarity, we have refrained from listing several thousand uncompressed TIFF files in the folders designated ‘Raw’.

Figure 1. Illustration of some of the ways in which subtle aspects of data organisation impact its usability.

These include: 1) long file/folder names with spaces, non-ASCII characters (ö) and redundant directories (ASCII – American Standard Code for Information Interchange); 2) obscure names or identifiers, possibly with inconsistent spelling, 3) inconsistency in folder hierarchy, 4) obscurity through meaningless names or identifiers, 5) verbosity in names, 6) subtle differences in spelling (in this case, a hyphen) and 7) inconsistency in typography due to character case and inclusion of different separator characters, e.g., spaces vs hyphens. See text for more details.

The example dataset illustrates several properties of its organisation that undermine the goal of being usable:

The 10 recommendations we present below are divided into four groups: planning (recommendation 1), structure (recommendations 2-4), naming (recommendations 5-7) and miscellaneous (recommendations 8-10). We have provided further guidance within each group for related concepts.

Miscellaneous

Finally, this section includes some tips on how to handle other aspects of organisation not covered in the previous sections.

(8) File formats.

• a. Provide images in widely used file formats unless you are demonstrating a novel file format in which case it may be necessary to first get in touch with the archive to plan accordingly. Additional information may be requested to provide users with guidelines on how to use and visualise the new format files including any conversion tools that are available or providing the same data in a widely used file format as well.

• b. Even for file types that are widely used, stick to open formats to ensure that users without access to proprietary software can access the data. Open formats promote the prevalence of tools (open source or proprietary) that can read and write data. We recommend the use of OME-NGFF (Moore et al., 2021) and OME-TIFF (Linkert et al., 2010) as open, widely supported imaging file formats.

(9) Document your data.

• a. Include a README text file which provides an overview of how the data is organised. Depositors may use it to discover the main facets by which the data is organised, the structure of any ad hoc text files as well as the meaning of naming entities used in file/folder names.

• b. Test the usability of your data by asking a colleague to peruse your data to assess whether the organisation is clear.

(10) Integrity. Include checksums, parity codes or hashes for each data file in a separate file, e.g., md5-sums.txt, imageset01.par2 or sha512-hashes.txt to facilitate content verification. These will allow users to verify that the data has not been corrupted during the deposition or download process. Each of these different ways to verify file integrity have corresponding tools available for all operating systems, but their operation is beyond the scope of this article (Lianhua & Xingquan, 2017).

Applying the recommendations above, we may revise the path:

data/A U Thör et al - A very long relevant title that has most of the keywords in your paper/A Folder with an overall description/0923480928 - Treatement Tr1-323 Tissue/0923480928_Treatement_Tr1323_Organelle1-topology1.zip’ is ‘0923480928 Treatement Tr1-323 Segmentation/0923480928_Treatement_Tr1323_Organelle1-topology1.zip

to:

data/brief_description/treatment3_tissue/segmentation/organelle1_topology1.zip

to achieve a reduction from 328 to 79 characters for the full path. The new organisation is presented in Figure 2.

Figure 2. Tree representation of the data from Figure 1 reorganised by applying some of the 10 recommendations described in this paper.

Conclusion

We hope that these 10 recommendations will only be the beginning of a broader discussion on how to organise bioimaging data in particular and experimental data in general for maximum usefulness, not just to the bioimaging community, but to the wider scientific community. Given the breadth of applications of bioimaging techniques, good organisation would go a long way to helping scientists from other disciplines to benefit from using bioimaging data. There is still considerable scope to develop better ways of not only organising data, but also representing it to enable automated data analysis.