Fiji plugins for qualitative image annotations: routine analysis and application to image classification

Implementation

We developed a set of Fiji plugins for the assignment of multiple pre-defined keywords to images. This set of keywords is entered by the user at the beginning of the annotation session. The image annotation is then carried out using an intuitive graphical user interface consisting of either buttons, checkboxes, or dropdown menus, which are automatically generated from the user-defined keywords (Figure 1–Figure 3). In addition to the pre-defined set of keywords, arbitrary image-specific comments can also be entered via a text input field. Furthermore, if run Measure is selected in the graphical interface, quantitative measurements as defined in Fiji’s menu Analyze>Set Measurements are reported in addition to the selected keywords. By default, the annotations and measurements are assigned to the entire image but can also describe image-regions outlined by regions of interest (ROI). The latter is simply achieved by drawing a new ROI on the image or selecting existing ROIs in the ROI Manager before assigning the keywords (Figure 3). Newly drawn ROIs are automatically added to the ROI Manager upon annotation with the plugins.

Figure 1. Single-category annotation of images in multi-dimensional stacks.

(A) Example of multi-dimensional image stack used to annotate mitotic stages in time-lapse data (source: ImageJ example image “Mitosis” – image credit NIH). (B) Graphical interface of the single class (buttons) plugin configured for annotation of 4 mitotic stages. (C) Results table with annotated categories (column category) generated by the plugin after selecting the single category column option in the plugin configuration window (not shown). (D) Alternative results table output using 1-hot encoding after selecting the option 1 column per category. The resulting 1-hot encoding of categories can be used for the training of classification algorithms.

Figure 2. Annotation of multiple categories using the multi-class (checkboxes) plugin.

(A) Example images of transgenic zebrafish larvae of the Tg(wt1b:egfp) transgenic line after injection with control morpholino (upper panel) or with ift172 morpholino (lower panel) inducing pronephric cysts. In this illustration, the plugin is used to score overall morphology and cyst formation. It could also be used to mark erroneous images (such as out-of-focus or empty wells). Images are from (Pandey et al., 2019). (B) Graphical interface of the checkbox annotation plugin configured with 2 checkboxes for overall morphology, 2 checkboxes for presence of pronephric cysts, and checkboxes to report out-of-focus and empty wells. Contrary to the single class (button) plugin, multiple categories can be assigned to a given image. (C) Resulting multi-category classification table with binary encoding of the annotations (True/False).

Figure 3. Qualitative and quantitative annotations of image regions using the multi-class (dropdown) plugin.

(A) ImageJ’s sample image “embryos” after conversion to grayscale using the command Image > Type > 32-bit (image credit: NIH). The embryos outlined with yellow regions of interest were annotated using the “multi-class (dropdown)” plugin. The insets at the top shows the annotation of overlapping ROIs, here corresponding to embryos with phenotype granular texture, dark pigmentation and elliptic shape. The inset at the bottom shows other embryos with different phenotypes (10: smooth/clear/circular, 12: granular/clear/elliptic, 14: smooth/dark/circular). (B) Graphical interface of the multi-class (dropdown) plugin. Three exemplary features are scored for each embryo: texture (granular, smooth), shape (circle, ellipse) and pigmentation (dark, clear). Quantitative measurements as selected in the Analyze > set Measurements menu (here Mean, Min and Max grey level) are also reported for each embryo, when the run Measure option is ticked. (C) ROI Manager with ROIs corresponding to annotated regions. (D) Resulting classification table with the selected features, qualitative measurement and associated ROI identifier for the outlined embryos.

The selected keywords, comments, measurements, and the image filename and directory are reported in a Fiji result table window with one row per annotation event (Figure 1C, D). An annotation event is triggered when a button is clicked, or a keyboard shortcut is pressed. With ROIs, the identifiers of the ROIs are also reported in the table, and the descriptors are saved as properties of the ROI objects. The information can then be retrieved from the ROIs using the Fiji scripting functions. The result table is updated row by row, as the user progresses with the annotations and can be saved as a csv file at any time. The csv file can be edited in any spreadsheet software or text editor, for instance to update the image directory column when images have been transferred to a different location or workstation.

Three plugins are provided to accommodate for different annotation modalities. The single-class (buttons) plugin (Figure 1) allows the assignment of a single keyword per image by clicking the corresponding button. With this plugin, the user can decide if the result table should contain a single category column containing the clicked category keyword for each image (Figure 1C), or one column per category with a binary code (0/1) depicting the assignment (Figure 1D). The latter option is particularly suitable for the training of supervised classification algorithms, which typically expect for their training an array of probabilities with 1 for the actual image-category and 0 for all other categories (also called 1-hot encoding, see (Müller & Guido, 2016)).

The multi-class (checkboxes) plugin (Figure 2) allows multiple keywords per image, which are selected via associated checkboxes (Figure 2B). This yields a result table with one column per keyword, and a 0/1 code if the keywords apply or not (Figure 2C). In this case, the table structure is similar to Figure 1D except that multiple keywords might be selected for a given image (i.e. multiple 1 for a given table row, as in row 1).

The multi-class (dropdown) plugin (Figure 3) allows to choose keywords from several lists of choices using dropdown menus (Figure 3B). The labels and choices for the dropdown menus are defined by the user in a simple csv file (Extended data, Supplementary Figure 1) (Thomas, 2020). This is convenient if multiple image features should be reported in separate columns (content, quality, etc.) with several options for each feature.

Operation

The plugins run on any system capable of executing Fiji. Executing one of the plugins will first display a configuration window requesting the user-defined keywords. Upon validation of the configuration, the actual annotation interface as in Figure 1–Figure 3 is displayed and ready to use for annotation. For the single-class (buttons) plugin, annotations can be recorded by clicking the corresponding category button, or by pressing one of the F1-F12 keyboard shortcuts. The shortcuts are automatically mapped to the categories in the order of their respective buttons, e.g. pressing F1 is equivalent to clicking the leftmost button. For the multi-class plugins, the annotations can be recorded by clicking the Add button or pressing one of the + keys of the keyboard.

For every plugin, an annotation event updates the results table as described in the implementation section, stores any newly drawn ROI into Fiji’s ROI Manager, and if the corresponding option is selected in the graphical user interface, the next image slice is displayed when a stack is annotated. The annotation table can be saved to a csv file at any time like a regular Fiji table. Moreover, the annotations can be appended to an existing result table in Fiji, provided the associated window is entitled “Annotations” or “Annotations.csv”.

Visualizing data-distribution with interactive sunburst chart

Sunburst charts offer an intuitive option for the visualization of the distribution of multiple qualitative features in a dataset. In these charts, the distributions of distinct features are represented as concentric “pies”, which help to identify relations between features. Supplementary Figure 2B (see extend data, Thomas, 2020) shows an example of such a sunburst chart generated with the provided workflow (Sunburst-chart-workflow) for the visualization of three qualitative features (texture, shape, and pigmentation) describing samples annotated in Figure 3. To generate such visualization, the workflow takes as input a table as in Figure 3 and counts the occurrence for each unique combination of qualitative features appearing in the table. The resulting occurrence-table (Extended data, Supplementary Figure 2A) (Thomas, 2020) is then represented as a sunburst chart. The order of the concentric circles is defined by the order of the columns in the annotation table, which can be configured in the workflow. In KNIME, the visualization is rendered as an interactive figure, allowing high-lighting specific sections of the plot, or reducing the number of displayed levels to inspect the distribution of a sub-population (e.g. samples with a smooth texture only). Finally, the provided workflow can be adapted for custom analysis and visualization, using any functionality available in KNIME.

Training a deep learning model for image classification

The plugins can also be used to generate the ground-truth annotation for the training of supervised image classification algorithms, such as deep learning models. For this purpose, we adapted an existing KNIME example workflow designed for the training of a convolutional neural network (CNN) by transfer learning, to directly use as input the annotation table provided by the single-class (buttons) plugin (Figure 4). The workflow generates a hybrid model architecture made of a pre-trained VGG16 base (Simonyan & Zisserman, 2015) used for feature-extraction from the images, completed by newly initialized fully-connected layers for the classification with custom classes. The workflow reads the image-locations and image-categories from the annotation table generated by the plugin, and performs classical data pre-processing for the training of CNNs: splitting the annotated dataset between training, validation and test sets, loading and pre-processing of the images and finally training and evaluation of the model. The trained model can be exported as a h5 file, suitable for the prediction of classes for new images. A second workflow is provided for this purpose. As an example of application, the workflow was used to train a model for the identification of two distinct morphological phenotypes of fluorescently-labelled pronephroi (normal and cystic) in transgenic zebrafish larvae as observed in a drug discovery screen (Pandey et al., 2019) (Figure 2A and Figure 4A). For this particular dataset and classification problem, the trained model readily achieved high classification accuracy (>99%, see confusion matrix Figure 4C) on a test set of 145 images (69 normal, 76 cystic) after training for only 5 epochs with a distinct training set of 116 images (57 normal, 59 cystic).

Usually training a custom deep learning model requires substantial programming skills, and a good comprehension of deep learning architecture and parameters. Recent efforts have aimed at facilitating the access of deep learning solutions to non-expert users (Gómez-de-Mariscal et al., 2019; Kräter et al., 2020; von Chamier et al., 2020); however, they usually do not cover the full pipeline from ground-truth annotation to model training, evaluation and application. Here we propose a complete pipeline for image classification that does not require major coding skills and is pre-configured with widely applicable default parameters. The workflow is expected to yield a model with satisfactory classification performance with the default parameters, such that non-expert users can directly get familiar with the method. For better performance, parameters for data pre-processing and model training (e.g. number of epochs, batch-size, learning rate) can be adapted in the configuration window of the Keras network learner node (Extended data, Supplementary Figure 3A). Importantly, using transfer-learning and only training the newly initialized fully-connected layers, the size of the training set and the time to train the network until it reaches a decent accuracy can be dramatically reduced compared to a training from scratch. For the classification of zebrafish pronephroi described above, the training by transfer learning took about 1 minute on a workstation with a Nvidia GTX760 GPU with 2 GB dedicated memory. In contrast, reaching a similar accuracy with the same model trained from scratch would likely require a few hours even on more modern and high-end hardware. Regarding hardware requirement, a CUDA-capable GPU is advised to achieve rapid model training, but the training can also be carried out with CPU only, on any machine supporting KNIME and python. Readers can refer to the GitHub repository (see section Software availability) for detailed descriptions of installation and requirements.

Underlying data

Zenodo: Fluorescently-labelled zebrafish pronephroi + ground truth classes (normal/cystic) + trained CNN model. https://doi.org/10.5281/zenodo.3997728 (Pandey et al., 2020).

This project contains the following underlying data:

Extended data

Zenodo: Qualitative image annotation plugins for Fiji - https://doi.org/10.5281/zenodo.4064118 (Thomas, 2020).

This project contains the following extended data:

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).