Ten recommendations for organising bioimaging data for archival

Introduction

Resources such as EMPIAR (Iudin et al., 2016; Iudin et al., 2023) and the BioImage Archive (Ellenberg et al., 2018; Hartley et al., 2022) provide a valuable service to the life-science community by supporting the archival and reuse of imaging data, often acquired at considerable cost, in line with the aspirations of the FAIR Guiding Principles (Wilkinson et al., 2016). There are numerous advantages and benefits to reusing bioimaging data, including more economical use of limited resources such as instrumentation and highly skilled technical staff. Moreover, specimens may be unique, costly to acquire, or difficult to reproduce, meaning that such data may only be accessible via archives. Archived data can be mined for reanalysis, verification and validation, and for development of new analytical techniques and software tools, such as machine learning model training. Reuse of such data may also lead to improvements in how it is produced, both technologically and methodologically. As practitioners in bioimaging data archiving, it is our experience that handling large datasets presents several data-management challenges, particularly in recent years with the rapidly increasing volumes of bioimaging data (Ellenberg et al., 2018). For example, it took eight years for EMPIAR to archive a total of one petabyte of data, but the second petabyte took only 14 months (Iudin et al., 2023). Bioimaging datasets may comprise numerous and sometimes very large files in a variety of, sometimes proprietary, formats. Individual files may include multiple channels and time points and data and metadata from several specimens. Besides the raw image data, there may also be a need to archive processed data, reconstructed 3D volumes, segmentations, particle stacks and other derived or related data.

There are two related but distinct avenues for organising data: labelling (metadata) and arranging data items (order). Metadata are essential to make the data useful even though metadata standards are difficult to enforce. Therefore, metadata standardisation has received a lot of attention with initiatives such as Bioschemas, an effort to improve findability of datasets via standardised textual annotations, and MIAME (Brazma et al., 2001), recommendations for minimal metadata describing a microarray experiment, and the overarching FAIR Guiding Principles (Wilkinson et al., 2016). For bioimaging, REMBI (Sarkans et al., 2021) provides community-supported recommendations on how to describe all aspects of bioimaging experiments including sample preparation, data processing and analysis. Whereas there are several ongoing efforts towards standardising bioimaging data formats (OME-NGFF (Moore et al., 2021), DVID (Katz & Plaza, 2019), HDF5 (Pietzsch et al., 2015), etc.), we know of no efforts towards harmonising how to organise datasets for maximum usefulness with archival in mind. The organisation (order) of data is usually taken for granted and it falls upon refinements of the metadata to bear the burden of meaningfully describing the data.

Motivation

Good organisation (order) of data improves its usefulness and is the responsibility of the data depositors. Depositors are best placed to present data in a way that adequately captures the experimental design and outcomes. Organising a dataset to minimally convey a structure in line with the actual experimental output can improve its usability while the bulk of meaningful attributes can be expressed in the metadata. The degree of usefulness depends directly on the quality of organisation, and thoughtful consideration of users’ needs improves that usefulness. Good organisation also gives a dataset transparency and understandability: users are able to immediately distinguish the various experimental categories as well as plan how to analyse the data. Therefore, it helps to have a clear perspective of the various classes of users.

In general, we consider three classes of users: intra-domain scientists, inter-domain scientists and extra-domain scientists (Datta et al., 2021). (For the purpose of this article, we will refer to any such user of a dataset as a ‘scientist’, interested in extracting some knowledge from the archived data.) Intra-domain scientists are familiar with key attributes of the data and may be able to quickly assess the usefulness of a dataset. An example would be a structural biologist mining an electron cryo-tomogram to extract sub-volumes that have not been previously studied. Inter-domain scientists may want to mine the data for purposes tangential to some other domain. For example, a genomicist may want to include structural analyses in a genomics study and may turn to raw imaging data to accomplish this. Extra-domain scientists are only interested in data for its technical properties, i.e., for some purpose completely unrelated to the original purpose of the data’s collection. A computer scientist, for example, may want to assess the performance of a learning algorithm on fluorescent microscopy images when performing some classification task. It is likely to be a challenge to optimise the organisation of data for all classes of users simultaneously. In practice, organising the data to be useful to scientists with the least familiarity with the domain will most likely advance its usefulness for all classes of scientists and can thus be a good aspiration.

The task of organising data consists of making trade-offs in the use of ‘ways and means’ of effecting the organisation. We will refer to these ‘ways and means’ as organisational resources. A simple example would be the use of alphabetic ordering when organising a set of strings; the natural ordering according to some alphabet is our organisational resource and we exploit the fact that most users will perceive this as a convenience when traversing the data. In less trivial organisational tasks, we need to express complex relationships between the entities to be organised. For instance, a dataset that consists of the experimental measurements resulting from a sequence of treatments on a set of specimens measured at various points in time requires the use of specimen, treatment and time point identifiers as well as other experimental aspects (data formats, alternative perspectives, transformations of the data such as changes in units, etc.) to be captured in such a way as to preserve the main experimental relationships. In that case, we can expand our set of organisational resources to include a folder hierarchy and file formats in addition to the set of symbols (letters, numerals, punctuation, literal symbols, uppercase and lowercase and so on) used to create the various identifiers. Ideally, we would like to keep repetition to a minimum so that the nature of the experiment can be readily discerned.

The manner in which organisational resources are used affects the usability of the resulting organisation: using too few of them will obscure the meaning of the organisation while using too many will overwhelm potential users. For example, including redundant folders along any part of the hierarchy (folders that contain only a single folder which in turn contains the actual data) makes it tedious to navigate through a dataset. On the other hand, dumping all files into one folder will make it difficult for the end user to distinguish between groups of semantically related files, especially when thousands of files are present. Similarly, naming files and folders by referring to entities inaccessible to its intended users (e.g., private machine names or private accession codes that external users will not have access to or even fathom) consumes precious ‘name space’ without conveying any useful information. Organising data is thus an investment of time and effort with the ultimate aim of improving the usefulness of the data.

We can therefore formulate the organisation task as follows: given a set of related data items associated with an experiment, how may they be organised to best convey their relationships using as few organisational resources as possible while maximising their usability?

To achieve this, we define the term facet to refer to the various attributes germane to the experiment which may be included in the folder and file names. A non-exhaustive list of facets are: specimen names (organism, tissue, cell type/line), experimental roles (treatments vs. controls), time (developmental status, date, elapsed time), processing status (raw data, by algorithm, procedure), generally available equipment (microscopes, detectors, preparation equipment model names), replicates, file types (3D volumes, particle stacks), names of software used for processing, and so on.

This guide attempts to solve the organisation task by providing 10 recommendations that arise from our experience of handling hundreds of large image datasets in the public archives EMPIAR, BioImage Archive and BioStudies (Sarkans et al., 2018). Ideally, we would like to organise potentially numerous and voluminous data to maximise ease of use so as to facilitate the user’s ability to:

This guide does not offer any recommendations for a detailed schema to describe experimental and analytical procedures; those may be captured in metadata for the various archives. Neither does it describe how to decide which experimental facets are appropriate (these are part of the experimental design), nor does it attempt to describe how to achieve organisation for automated analysis (we assume that the resulting organisation will be consumed by humans). It also ignores the universe of image formats in use and mainly includes examples from our experience archiving bioimaging data, but we anticipate it may be useful across other imaging disciplines. Good organisation improves data structure and format predictability and may facilitate automated processing. Therefore, our guide is intended to lead towards best practices rather than serve as a framework. Finally, this guide does not aim to achieve standardisation. We believe it is more practical to have a set of best practices and leave it up to the data authors to decide how best to apply them.

We believe that the recommendations outlined here may be of value to two principal groups of users: 1) data depositors, who need to design and prepare their data to improve its usability to the community, and 2) technologists (hardware, software and methods developers), who, by considering these recommendations in their designs, can greatly facilitate data organisation at the source.

To make our recommendations practical, we have developed bandbox (Korir et al., 2022), an open-source command-line interface (CLI) tool to help users understand how they can improve the organisation of their data in preparation for archival. The program offers two CLI commands: view and analyse. The view command displays a tree of a directory and all its contents; for every non-empty directory with files, bandbox provides a summary of the number of files in it, including a list of all the file formats encountered. The analyse command provides a listing of possible issues grouped into categories in line with those specified in the Recommendations section of this article. bandbox examines the tree associated with the nested hierarchy of files and folders in a dataset and then concurrently runs various heuristics on the tree which are controlled by configurations that the user may modify. The results produced by the analyse command are only suggestions for improvement; we understand that there may be practical limitations to implementing some of the suggested improvements as well as good reasons for keeping the data as is. Figures 1 and 2 show screenshots of the results of running bandbox on two different datasets.

Figure 1. Example of a dataset with organisational red flags identified by bandbox such as use of spaces or non-ASCII characters, redundant directories and so on with an indication of the number of such entities found.

The actual example dataset is provided with the bandbox source code (ASCII – American Standard Code for Information Interchange).

Figure 2. Example of a dataset with no red flags as inferred by bandbox.

The actual example dataset is provided with the bandbox source code.

Recommendations

We will motivate our guide by referring to a fictitious EMPIAR dataset. This dataset has a clear structure, but we propose that it can be further improved following the recommendations in the guide below.

Our goal is to improve the file/folder structure shown in Figure 3 to better convey the relationships between the experimental facets while economising the organisational resources available. For clarity, we have refrained from listing several thousand raw TIFF files in the folders designated ‘Raw’.

Figure 3. Illustration of some of the ways in which subtle features of data organisation impact its usability.

Issues include: 1) long file/folder names with spaces, non-ASCII characters (ö) and redundant directories (ASCII – American Standard Code for Information Interchange); 2) obscure sequences with inconsistent spelling, 3) inconsistency in folder hierarchy, 4) obscurity through meaningless symbol sequences, 5) verbosity in names, 6) subtle differences in spelling (in this case, a hyphen) and 7) inconsistency in typography due to character case and inclusion of different separator characters, e.g., spaces vs hyphens. See text for more details.

The example dataset illustrates several properties of its organisation that undermine the goal of being usable:

The 10 recommendations we present below are divided into four groups: planning (one recommendation), structure (three recommendations), naming (three recommendations) and miscellaneous (three recommendations). We have provided further guidance within each group for related concepts.

Miscellaneous

Finally, this section includes some tips on how to handle other aspects of organisation not covered in the previous sections.

(8) Friendly file formats.

• a. Consider providing images in widely used file formats unless you are demonstrating a novel file format in which case it may be necessary to first get in touch with the archive to plan this. Additional information may be requested to provide users with guidelines on how to use and visualise the new format files including any conversion tools that are available and on providing the same data in a widely used file format as well.

• b. Even for file types that are widely used, it may be helpful to stick to open formats to ensure that users without access to proprietary software will have access to the data.

(9) Document your data.

• a. Consider including a README text file which provides an overview of how the data is organised.

• b. Consider testing the usability of your data by asking a colleague to peruse your data to assess whether the organisation is clear. This can be achieved by asking the tester to describe their understanding of what the data presents.

(10) Integrity. If possible, consider including checksums, parity codes or hashes for each data file in a separate file, e.g., md5-sums.txt, imageset01.par2 or sha512-hashes.txt to facilitate content verification. These will allow users to verify that the data has not been corrupted during the deposition or download process. Each of these different ways to verify file integrity have corresponding tools available for all operating systems, but their operation is beyond the scope of this article (Chi Lianhua & Zhu Xingquan, 2017).

Applying the recommendations above, we may revise the path:

data/A U Thör et al - A very long relevant title that has most of the keywords in your paper/A Folder with an overall description/0923480928 - Treatement Tr1-323 Tissue/0923480928_Treatement_Tr1323_Organelle1-topology1.zip’ is ‘0923480928 Treatement Tr1-323 Segmentation/0923480928_Treatement_Tr1323_Organelle1-topology1.zip

to:

data/brief_description/treatment3_tissue/segmentation/organelle1_topology1.zip,

a reduction from 328 to 79 characters for the full path. The new organisation is presented in Figure 4.

Figure 4. Tree representation of the data from Figure 3 reorganised by applying some of the 10 recommendations proposed.

Conclusion

We hope that these 10 recommendations will only be the beginning of a broader discussion on how to organise bioimaging data in particular and experimental data in general for maximum usefulness, not just to the bioimaging community, but to the wider scientific community. Given the breadth of applications of bioimaging techniques, good organisation would go a long way to helping scientists from other disciplines to benefit from using bioimaging data. There is still considerable scope to develop better ways of not only organising data, but also representing it to enable automated data analysis.