DeepCLEM: automated registration for correlative light and electron microscopy using deep learning

In correlative light and electron microscopy (CLEM), the fluorescent images must be registered to the EM images with high precision. Due to the different contrast of EM and fluorescence images, automated correlation-based alignment is not directly possible, and registration is often done by hand using a fluorescent stain, or semi-automatically with fiducial markers. We introduce “DeepCLEM”, a fully automated CLEM registration workflow. A convolutional neural network predicts the fluorescent signal from the EM images, which is then automatically registered to the experimentally measured chromatin signal from the sample using correlation-based alignment. The complete workflow is available as a Fiji plugin and could in principle be adapted for other imaging modalities as well as for 3D stacks.


Introduction
Correlative Light and Electron Microscopy (CLEM) combines the high resolution of electron microscopy (EM) with the molecular specificity of fluorescence microscopy.In superresolution array tomography (srAT) for example, serial sections are imaged first under the fluorescence microscope using super-resolution techniques such as structured illumination microscopy (SIM), and then in the electron microscope 1 .With this technique, it is possible to identify and assign molecular identities to subcellular structures such as electrical synapses 1,2 or microdomains in bacterial membranes 3 that cannot be resolved by EM due to insufficient contrast.
To visualize and interpret the results of CLEM, the fluorescent images must be registered to the EM images with high accuracy and precision.Due to the different contrasts of EM and fluorescence images, automated correlation-based image alignment, as used e.g. for aligning EM serial sections 4 , is not directly possible.Registration is often done by hand using a fluorescent chromatin stain 2 , or semi-automatically with fiducial markers using tools such as eC-CLEM 5 .Further improvement and automation of the registration process is of great interest to make CLEM scalable to larger datasets.
Deep Learning using convolutional neural networks (CNNs) has become a powerful tool for various tasks in microscopy, including denoising and deconvolution as well as classification and segmentation, reviewed in 6 and 7.One interesting application of CNNs is the prediction of fluorescent labels from transmitted light images of cells, also called "in silico labeling" 8,9 .
We show here that this approach can be used to predict the fluorescent chromatin stain in electron microscopy images of cell nuclei.The predicted "in silico" chromatin images are sufficiently similar to real experimental chromatin images acquired with SIM to use them for automated correlation-based registration of CLEM images.Based on this observation, we developed "DeepCLEM", a fully automated CLEM registration workflow implemented in Fiji 10 and based on CNNs.

Data acquisition
We used previously acquired imaging data of Caenorhabditis elegans and of human skin samples from healthy subjects.Sample preparation as well as the acquisition of the imaging data has been previously described in detail 1,2,11 .Briefly, C. elegans worms were cryo-immobilized via high-pressure freezing and subsequently processed by freeze substitution.All samples were embedded in methacrylate resin and sectioned at 100 nm.Ribbons of consecutive sections were attached to glass slides and labeled with fluorophores.Live Hoechst 33342 was used to stain chromatin and immunolabeling was used to visualize molecular identities.The sections were then imaged with SIM super-resolution microscopy.Next, they were processed for electron microscopy by heavy metal contrasting and carbon coating.The regions of interest previously imaged with SIM were then imaged again on the same sections with scanning electron microscopy, resulting in pairs of images that needed to be correlated.

Manual registration
To prepare ground truth for network training, we manually registered the chromatin channel to the EM images as described in 2. We selected 30 subimages and super-imposed them in the software Inkscape.By reducing the opacity of the chromatin images, they could be manually resized, rotated and dragged until the Hoechst signal coincided with the electron-dense heterochromatin puncta in the underlying EM images.To generate own training data, reproducible methods retaining a record of all transforms are recommended.

Implementation
We implemented DeepCLEM as a Fiji 10 plugin, using CSBDeep 12 for network prediction.Preprocessing of the images as well as network training were performed in Python using scikit-image 13 and TensorFlow 14 .First, a neural network trained on manually registered image pairs predicts the fluorescent chromatin signal from previously unseen EM images (Figure 1A).This "virtual" fluorescent chromatin image is then automatically registered to the experimentally measured chromatin signal from the sample using the "similarity" transform of the "Register Virtual Stack Slices" plugin in Fiji (Figure 1B).The transformation parameters from this automated alignment are finally used to register the other SIM images that contain the signals of interest to the EM image (Figure 1C).
We trained the ProjectionCARE 2D UNet architecture from CSBDeep 12 with 3 layers and a batch size of 8 for 150 epochs, using the default parameters of CSBDeep unless stated otherwise.Training on a GPU took between one (RTX 2080 Ti, cuda 11.3, Windows 10) and three hours (Tesla P100, cuda 11.7, Ubuntu 20.04) and around 10 hours on a CPU (Intel i7-8700K at 3.70 GHz, Windows 10).The Jupyter notebook for training on custom datasets as well as documentation how to use it are available online (see software availability statement).

Amendments from Version 2
We added information on the training parameters, the hardware and the training time required to the methods section of the manuscript.The software availability section now contains details on how to contact the authors for support and questions.The documentation of the software was expanded to include more details especially for novice users, with additional information about when and how the method should be adapted to novel data by training a custom network model.The installation of the python environment was updated and tested on different systems.

Any further responses from the reviewers can be found at the end of the article
Operation DeepCLEM requires Fiji 10 with CSBDeep 12 to run.The paths to the images and model file are entered in a user dialog (Figure 2).After running DeepCLEM, the correlated images and a .XML file containing the transform parameters are written to the output directory.The workflow is summarized in Figure 1; instructions for installing and running DeepCLEM are included in the repository.The network included in DeepCLEM was trained on in-house data and may work on images with similar contrast, but in most cases, re-training will be necessary -details on the workflow and training parameters are given in a Jupyter notebook in the repository.Running this notebook on a directory with 30-40 aligned ground truth image pairs will yield a model file that can be loaded in the DeepCLEM Fiji plugin.

Comparison of network architectures
We trained DeepCLEM on correlative EM and SIM images of C. elegans and on human skin tissue and compared prediction and registration results for different network architectures and preprocessing routines.A generative adversarial network (pix2pix) showed promising results in some images from the skin dataset, but overall performance was best using the ProjectionCARE network from CSBDeep 12 .
Optimization of preprocessing EM images had large differences in contrast even when acquired in the same laboratory.We compared different preprocessing routines, including normalization and histogram equalization, and found that standard histogram equalization in Fiji resulted in the best performance on our data.The best combination of preprocessing steps for optimizing contrast may however depend on the data.

Quantitative evaluation
We performed a quantification of the quality of the registration on four manually aligned images from an independent experiment, and applying a known shift or rotation.In 75% of cases the registration worked and had a very small error, while in 25% it was completely off by several 100 nm (Table 1).If two of the test images were included in the training set, the error was much lower, so DeepCLEM works best if a small number of images of each experiment are manually aligned and added to the training data.The remaining images are then reliably aligned.We also varied the number of images in the training set and found that 30-40 ground truth images are sufficient to obtain good alignment on the test set.

Discussion
We developed "DeepCLEM", a fully automated CLEM registration workflow implemented in Fiji 10 based on prediction of the chromatin stain from EM images using CNNs.Our registration workflow can easily be included in existing CLEM routines or adapted for imaging methods other than srAT where corresponding 2D slices need to be registered.If direct prediction of one modality from the other does not work, an alternative is to predict a common representation of both modalities, as described in Ref 15.While we found that "DeepCLEM" performs well under various conditions, it has some limitations: using chromatin staining for correlation requires the presence of at least three heterochromatin patches in the field of view.This limitation could be overcome by using e.g.propidium iodide to label the overall structure of the tissue.Widefield microscopy could be used where SIM is not available, but alignment quality is bounded by the lower-resolution channel.
The popular CLEM registration tool eC-CLEM 5 has an "autofinder" function that detects corresponding features using spot finding or segmented regions.We did not perform a direct comparison, but results should be similar if suitable spots are found.If not, then image-to-image translation with DeepCLEM followed by point-based registration in eC-CLEM could be a promising alternative.The authors present an image registration workflow to register light microscopy (LM) images into electron microscopy (EM) images for correlative light-electron microscopy (CLEM).To solve the multimodal registration problem, DeepCLEM predicts a LM image from the EM image using ProjectionCARE.The predicted LM image is then used to compute the necessary transformation of the LM image to the EM image.Apart from the pre-trained model, which is limited to data it has been trained on, the authors also describe how to train the model on their own data.
Overall, a very nice approach that has been well implemented.
In my review I verified if the provided software package and its documentation allows the successful reproduction of the described workflow.I also assessed if the provided article sufficiently describes and discusses the proposed approach (i.e.scientific context and limitations).

Application of the pre-trained model:
I was able to easily install the software as per documentation.The provided test images allow to easily verify the correct operation of the Fiji workflow.The output data is clearly and sufficiently described.Positive is that the transformation file is provided that allows to apply the transformation on other data.

Retraining:
Great that the conda .yml is provided for easy installation of the dependencies.As a warning, I experienced that the creation of the conda environment took a long time on my machine.But could be specific to my setup.The possibilities of my review here are also limited since I do not have the possibility to generate my own training data from scratch and I can only test the provided data.Within these constraints, I was able to test the notebooks for their correct operation.

Article:
The article is overall well written and explains the context of the work.The authors also test and discuss the limitations of the approach.Further, providing avenues to overcome problems and limitations.
As a side note, not relevant to the review, in terms of the registration field in the biology focused sciences, I am always a bit surprised that software solutions such as elastix (https://elastix.lumc.nl/)are not more widely used and discussed.I would have expected that the solutions of multimodal image registration developed in the medical field would be at least somewhat relevant to CLEM.The article seems to also discuss another established registration tool (eC-CLEM) so there is sufficient other context for this publication.
The methods section of the article is unfortunately a bit sparse.The actual network architecture is only mentioned in the results.No further detail is provided.It is fine if the default values of the out-of-the-box network are used.However, this should be clearly stated in the methods.I would also wish a better description of how the training has been performed.Including how long the training took and on which hardware this was performed.
I really enjoyed that the authors provide all necessary code and example images on their GitHub repo.The only tiny thing missing is to clearly state their preferred way of users getting in touch for support.I would suggest clearly stating this in the software availability section.
I endorse the article under the following conditions: A more extensive methods section.Describing clearer what the architecture was and any important or changed hyperparameters.Also, the hardware and the training time should be specified.

○
Improve documentation from the perspective of a novice user.A few questions arise when reading the manuscript -which do not modify the overall positive impression of this study.
The group uses Hoechst 33342 for staining and the biological objects are visualized by SIMwould it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere? 1.
What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images?Does this macro work as provided, i.e. 'fully automatically'?Is any user intervention needed?-the authors trained their macro on their datasets.Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase?if so: workflow? 2.
Histogram equalization of EM images -was this done with routines implemented in Fiji? 3.
Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM?or at least comment on this?

4.
With comments on my points listed above, and according improvements in the manuscript, this can be indexed -thus, approved with reservations.

Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Partly
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: TEM, STEM, tomography, CLEM; microbiology, cell biology I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 28 Jul 2022

R: The group uses Hoechst 33342 for staining and the biological objects are visualized by SIMwould it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere?
A: If the structures used for alignment (e.g., chromatin) can be resolved and the WF images can be manually aligned, it is also possible to obtain a prediction and do automated alignment, although the achievable quality of the registration is obviously limited by the resolution of the fluorescence channel.We now added this in the discussion of the scope and limitations of the tool.

R: What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed?
A: Using a trained network for prediction and alignment can be done fully automatically in Fiji, only the paths of the images and network must be provided in the user dialog.For retraining, some knowledge (e.g., setting up a python environment) is necessary.We now added a more detailed description of the training workflow.Currently, the training process should probably be performed by some more experienced person while using the trained network does not require any special knowledge.

R: The authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow?
A: It may be possible to use the provided network trained on our data on images from other labs directly, however in most cases re-training is required.We now describe the required workflow in more detail and perform a quantitative comparison of registration quality where images from an experiment were either included in the training set or not.This type of effort is crucial in harnessing the huge potential of correlative multimodal imaging as cross-modal registration is a significant bottleneck in many cases.The method of imaging both LM and EM on the processed and cut sections is perhaps not as widespread as other methods, such as, e.g.whole cell confocal imaging followed by resin-embedding, followed by EM imaging.The method used in this paper avoids the familiar CLEM registration problems of non-linear warping induced by the EM sample preparation and mismatched axial resolutions.This constrains the registration task to a "similarity" type transform rather than affine or non-rigid as is required in some other CLEM registration tasks.
The open source plugin is easy to install and runs as described on the test data provided.
A few points that I think should be addressed: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" -this seems overly specific, in general CLEM alignment might make use of many different markers or features.Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.

○
The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin.Is there an additional *.ijm IJ1 macro available, or is this a typo?○ Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element.It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.
○ I believe the preferred capitalisation is "Fiji" and not "FIJI" ○ ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains.ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool.It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.

○
For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms.Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.

○
In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.

○
If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for nonspecialists.

○
The text mentions "correlation based alignment in Fiji" -looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow.Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.

○
An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually.Understanding this is likely to be very important for researchers who might be considering using this tool.If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial!The authors propose extensions to their method for dealing with situations where nuclei are not present in the images.It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it.For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli -this should be mentioned in the body of the paper as well.
○ Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face.To avoid potential confusion, this constraint should be made more clear in the abstract.

○
The abstract and discussion mention adaptations to apply to other modalities and 3D stacks.Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful ○ Overall, the tool itself works as described on the data type provided, but I think there is a risk that users might be tempted to stretch beyond the current scope of the tool, so this scope should be made more clear in the text.Reviewer Expertise: Image Analysis, CLEM, Machine Learning, Electron microscopy I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Is
Author Response 28 Jul 2022 Philip Kollmannsberger R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" -this seems overly specific, in general CLEM alignment might make use of many different markers or features.Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.
A: We removed the mention of a specific stain in the abstract to reflect the general applicability of the method.

R:
The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin.Is there an additional *.ijm IJ1 macro available, or is this a typo?
A: The code is a Jython script with graphical user interface to be run within Fiji.From the user's perspective, it is indeed a plugin.We changed "macro" to "plugin", as suggested.R: Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element.It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.
A: The trained network can be applied to different datasets if they have similar contrast, usually from the same microscope or sample type.Otherwise, (re-)training of the network is necessary.We now make this clearer in the manuscript and include a description of the training workflow.

R: I believe the preferred capitalisation is "Fiji" and not "FIJI"
A: Thank you, we corrected this.R: ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains.ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool.It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.
A: We now refer to ec-CLEM autofinder which is similar in terms of functionality.It is based on finding corresponding features in both modalities, e.g., using wavelet-based spot detection or the centers of segmented regions, and then performing registration of these automatically detected landmarks.We did not perform a direct quantitative comparison.If suitable corresponding spots can be found (e.g.beads), the performance on the same images using the same registration algorithm should be comparable.In cases where spot finding or similar methods in ec-CLEM do not perform well because the images are too dissimilar or if there are no fiducial markers, then transferring the EM image to fluorescence using DeepCLEM could be used to generate input data for point-based registration in ec-CLEM.R: For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms.Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.
A: We agree that these methods are more reproducible and added an additional note to the paragraph on manual registration to encourage users to use such reproducible methods.Since we already had created the training data, we did, however, not repeat the alignment.For assessing the performance (see below), we instead artificially displaced the two channels -in this case, the steps and transforms are also known exactly.

R:
In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.
A: We now include more detail about the training workflow in the implementation section and compared the performance for different amounts of training images.R: If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists.
A: We extended both the description of the training parameters as well as the documentation of the entire workflow, including training in the Jupyter notebook.R: The text mentions "correlation based alignment in Fiji" -looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow.Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.
A: The text now explicitly mentions the use of the "Register Virtual Stack Slices" plugin and the "similarity" transform in our work, but in principle, our workflow should also work with other means of correlation-based registration.
R: An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually.Understanding this is likely to be very important for researchers who might be considering using this tool.If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com

Figure 1 .
Figure 1.Schematic of the "DeepCLEM" workflow.From the EM image (A), a CNN predicts the chromatin channel (B), to which the SIM image (C) is registered (D).The same transform is applied to the channel of interest (E) to obtain a CLEM overlay (F).

○
State how users should get in touch for support.○Is the rationale for developing the new software tool clearly explained?YesIs the description of the software tool technically sound?YesAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?PartlyIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?YesAre the conclusions about the tool and its performance adequately supported by the findings presented in the article?Yes Competing Interests: No competing interests were disclosed.Reviewer Expertise: Data science, bioimage analysis, cell biology.I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.Reviewer Report 30 August 2022 https://doi.org/10.5256/f1000research.136822.r146290ReinhardRachelCentre for Electron Microscopy, Faculty of Biology and Preclinical Medicine, University of Regensburg, Regensburg, GermanyThe manuscript by Seifert et al. (Univ.Würzburg, Germany) provides a convenient software tool (a macro name DeepCLEM) for linking fluorescent light microscopy images (here: structured illumination microscopy) to the information seen in electron micrographs (in this particular case: FE-SEM), obtained from serial sections on slides.In a first glance, this short paper describes the development and application of a flexible and easy to handle software tool to a program which is open to many microscopy researchers, ImageJ / Fiji, in form of a macro, which can easily be added to any Fiji installation.Thus, it appears that this tool can be used by many groups aiming to analyze biological samples using correleative light and electron microscopy (CLEM).
the rationale for developing the new software tool clearly explained?YesIs the description of the software tool technically sound?PartlyAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?PartlyIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?YesAre the conclusions about the tool and its performance adequately supported by the findings presented in the article?Partly Competing Interests: No competing interests were disclosed.

Table 1 .
Quantitative evaluation.When applying DeepCLEM on images from a different experiment not represented in the training data, registration failed in 25% of cases (top part, image 1-4).If two manually aligned images were included in the training set, all other test images were successfully registered (bottom part, images 1-2).