DeepCLEM: automated registration for correlative light and electron microscopy using deep learning

Rick Seifert; Sebastian M. Markert; Sebastian Britz; Veronika Perschin; Christoph Erbacher; Christian Stigloher; Philip Kollmannsberger

doi:10.12688/f1000research.27158.2

Home Browse DeepCLEM: automated registration for correlative light and electron...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Software Tool Article

Revised

DeepCLEM: automated registration for correlative light and electron microscopy using deep learning

[version 2; peer review: 1 approved, 2 approved with reservations]

Rick Seifert^1,2, Sebastian M. Markert², Sebastian Britz², [...] Veronika Perschin², Christoph Erbacher³, Christian Stigloher², Philip Kollmannsberger ¹

Rick Seifert^1,2, Sebastian M. Markert², [...] Sebastian Britz², Veronika Perschin², Christoph Erbacher³, Christian Stigloher², Philip Kollmannsberger ¹

PUBLISHED 02 Aug 2022

Author details Author details

¹ Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany
² Imaging Core Facility, Biocenter, University of Würzburg, Würzburg, 97074, Germany
³ Department of Neurology, University of Würzburg, Würzburg, 97074, Germany

Rick Seifert
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Software, Validation, Writing – Original Draft Preparation, Writing – Review & Editing

Sebastian M. Markert
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Sebastian Britz
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Veronika Perschin
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Christoph Erbacher
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Christian Stigloher
Roles: Conceptualization, Data Curation, Funding Acquisition, Investigation, Methodology, Resources, Supervision, Writing – Review & Editing

Philip Kollmannsberger
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Software, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Artificial Intelligence and Machine Learning gateway.

This article is included in the NEUBIAS - the Bioimage Analysts Network gateway.

Abstract

In correlative light and electron microscopy (CLEM), the fluorescent images must be registered to the EM images with high precision. Due to the different contrast of EM and fluorescence images, automated correlation-based alignment is not directly possible, and registration is often done by hand using a fluorescent stain, or semi-automatically with fiducial markers. We introduce “DeepCLEM”, a fully automated CLEM registration workflow. A convolutional neural network predicts the fluorescent signal from the EM images, which is then automatically registered to the experimentally measured chromatin signal from the sample using correlation-based alignment. The complete workflow is available as a Fiji plugin and could in principle be adapted for other imaging modalities as well as for 3D stacks.

Keywords

Correlative Microscopy, Image Registration, In-silico labeling, Deep Learning

Corresponding author: Philip Kollmannsberger

Competing interests: No competing interests were disclosed.

Grant information: SMM was supported by a PhD grant from the Studienstiftung des Deutschen Volkes (German Academic Scholarship Foundation). VP and CS are funded by Deutsche Forschungsgemeinschaft GRK2581 SPHINGOINF and CS by STI 700/1-1. Computational work was performed using the High-Performance Computing Cloud of Würzburg University (DFG project 327497565).This publication was supported by COST Action NEUBIAS (CA15124), funded by COST (European Cooperation in Science and Technology). The JEOL JSM-7500F field emission scanning
electron microscope (SEM) was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 218894895 (INST 93/761-1 FUGG) and the structured illumination
microscope (SIM) was funded by DFG - 261184502 (INST 93/823-1 FUGG). VP and CS were funded by DFG within RTG 2581 SphingoINF (project 6).

Copyright: © 2022 Seifert R et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Seifert R, Markert SM, Britz S et al. DeepCLEM: automated registration for correlative light and electron microscopy using deep learning [version 2; peer review: 1 approved, 2 approved with reservations]. F1000Research 2022, 9:1275 (https://doi.org/10.12688/f1000research.27158.2) First published: 27 Oct 2020, 9:1275 (https://doi.org/10.12688/f1000research.27158.1) Latest published: 28 Dec 2023, 9:1275 (https://doi.org/10.12688/f1000research.27158.3)

Revised Amendments from Version 1

TRAINING PROCESS:
We added more detail about the training workflow and discuss when retraining is necessary. The Jupyter training notebook now contains a description of training parameters and more documentation. We varied the number of training images and compared the resulting performance.

ERROR QUANTIFICATION:
We performed a quantification of the registration quality using manually aligned ground truth data as reference. If a few images from an experiment are included in the training set, the registration of the remaining images improves significantly (Table 1).

COMPARISON:
We now refer to ec-CLEM autofinder, which uses automatically detected landmarks for registration, but we did not perform a direct quantitative comparison. If suitable spots can be found, the performance on the same images should be similar. If not, then transferring the EM image to fluorescence using DeepCLEM could be used to generate input data for point-based registration in ec-CLEM.

SCOPE AND LIMITATIONS:
We added more detail regarding scope and limitations of this tool and its current requirements to abstract, introduction and discussion, including the need for at least three nucleoli if chromatin staining is used. To reflect the general applicability of the method, we replaced "fluorescent chromatin stain" by "fluorescent stain" in the abstract. We mention that widefield can be used, although with lower alignment precision, and that our tool could be used for other imaging methods if pairs of 2D slices are available.

MINOR CHANGES:
Changed "macro" to "plugin", "FIJI" to "Fiji". Added the information that standard histogram equalization in Fiji was used for preprocessing, and "Register Virtual Stack Slices" and similarity transform for registration. We encourage the reader to use more reproducible methods for generating training data.

See the authors' detailed response to the review by Christopher Schmied
See the authors' detailed response to the review by Reinhard Rachel
See the authors' detailed response to the review by Martin L. Jones

Introduction

Correlative Light and Electron Microscopy (CLEM) combines the high resolution of electron microscopy (EM) with the molecular specificity of fluorescence microscopy. In super-resolution array tomography (srAT) for example, serial sections are imaged first under the fluorescence microscope using super-resolution techniques such as structured illumination microscopy (SIM), and then in the electron microscope¹. With this technique, it is possible to identify and assign molecular identities to subcellular structures such as electrical synapses^1,2 or microdomains in bacterial membranes³ that cannot be resolved by EM due to insufficient contrast.

To visualize and interpret the results of CLEM, the fluorescent images must be registered to the EM images with high accuracy and precision. Due to the different contrasts of EM and fluorescence images, automated correlation-based image alignment, as used e.g. for aligning EM serial sections⁴, is not directly possible. Registration is often done by hand using a fluorescent chromatin stain², or semi-automatically with fiducial markers using tools such as eC-CLEM⁵. Further improvement and automation of the registration process is of great interest to make CLEM scalable to larger datasets.

Deep Learning using convolutional neural networks (CNNs) has become a powerful tool for various tasks in microscopy, including denoising and deconvolution as well as classification and segmentation, reviewed in 6 and 7. One interesting application of CNNs is the prediction of fluorescent labels from transmitted light images of cells, also called “in silico labeling”^8,9.

We show here that this approach can be used to predict the fluorescent chromatin stain in electron microscopy images of cell nuclei. The predicted “in silico” chromatin images are sufficiently similar to real experimental chromatin images acquired with SIM to use them for automated correlation-based registration of CLEM images. Based on this observation, we developed “DeepCLEM”, a fully automated CLEM registration workflow implemented in Fiji¹⁰ and based on CNNs.

Methods

Data acquisition

We used previously acquired imaging data of Caenorhabditis elegans and of human skin samples from healthy subjects. Sample preparation as well as the acquisition of the imaging data has been previously described in detail^1,2,11. Briefly, C. elegans worms were cryo-immobilized via high-pressure freezing and subsequently processed by freeze substitution. All samples were embedded in methacrylate resin and sectioned at 100 nm. Ribbons of consecutive sections were attached to glass slides and labeled with fluorophores. Live Hoechst 33342 was used to stain chromatin and immunolabeling was used to visualize molecular identities. The sections were then imaged with SIM super-resolution microscopy. Next, they were processed for electron microscopy by heavy metal contrasting and carbon coating. The regions of interest previously imaged with SIM were then imaged again on the same sections with scanning electron microscopy, resulting in pairs of images that needed to be correlated.

Manual registration

To prepare ground truth for network training, we manually registered the chromatin channel to the EM images as described in 2. We selected 30 subimages and super-imposed them in the software Inkscape. By reducing the opacity of the chromatin images, they could be manually resized, rotated and dragged until the Hoechst signal coincided with the electron-dense heterochromatin puncta in the underlying EM images. To generate own training data, reproducible methods retaining a record of all transforms are recommended.

Implementation

We implemented DeepCLEM as a Fiji¹⁰ plugin, using CSBDeep¹² for network prediction. Preprocessing of the images as well as network training were performed in Python using scikit-image¹³ and TensorFlow¹⁴. First, a neural network trained on manually registered image pairs predicts the fluorescent chromatin signal from previously unseen EM images (Figure 1A). This "virtual" fluorescent chromatin image is then automatically registered to the experimentally measured chromatin signal from the sample using the “similarity” transform of the “Register Virtual Stack Slices” plugin in Fiji (Figure 1B). The transformation parameters from this automated alignment are finally used to register the other SIM images that contain the signals of interest to the EM image (Figure 1C).

Figure 1. Schematic of the "DeepCLEM" workflow.

From the EM image (A), a CNN predicts the chromatin channel (B), to which the SIM image (C) is registered (D). The same transform is applied to the channel of interest (E) to obtain a CLEM overlay (F).

Operation

DeepCLEM requires Fiji¹⁰ with CSBDeep¹² to run. The paths to the images and model file are entered in a user dialog (Figure 2). After running DeepCLEM, the correlated images and a .XML file containing the transform parameters are written to the output directory. The workflow is summarized in Figure 1; instructions for installing and running DeepCLEM are included in the repository. The network included in DeepCLEM was trained on in-house data and may work on images with similar contrast, but in most cases, re-training will be necessary – details on the workflow and training parameters are given in a Jupyter notebook in the repository. Running this notebook on a directory with 30-40 aligned ground truth image pairs will yield a model file that can be loaded in the DeepCLEM Fiji plugin.

Figure 2. GUI and input parameters for "DeepCLEM".

Results

Comparison of network architectures

We trained DeepCLEM on correlative EM and SIM images of C. elegans and on human skin tissue and compared prediction and registration results for different network architectures and preprocessing routines. A generative adversarial network (pix2pix) showed promising results in some images from the skin dataset, but overall performance was best using the ProjectionCARE network from CSBDeep¹².

Optimization of preprocessing

EM images had large differences in contrast even when acquired in the same laboratory. We compared different preprocessing routines, including normalization and histogram equalization, and found that standard histogram equalization in Fiji resulted in the best performance on our data. The best combination of preprocessing steps for optimizing contrast may however depend on the data.

Quantitative evaluation

We performed a quantification of the quality of the registration on four manually aligned images from an independent experiment, and applying a known shift or rotation. In 75% of cases the registration worked and had a very small error, while in 25% it was completely off by several 100 nm (Table 1). If two of the test images were included in the training set, the error was much lower, so DeepCLEM works best if a small number of images of each experiment are manually aligned and added to the training data. The remaining images are then reliably aligned. We also varied the number of images in the training set and found that 30-40 ground truth images are sufficient to obtain good alignment on the test set.

Table 1. Quantitative evaluation.

When applying DeepCLEM on images from a different experiment not represented in the training data, registration failed in 25% of cases (top part, image 1–4). If two manually aligned images were included in the training set, all other test images were successfully registered (bottom part, images 1–2).

Absolute error [nm]
	aligned		shift in X (3125 nm)		rotation 90°		rotation 180°
	X	Y	X	Y	X	Y	X	Y
No images from same experiment included in training:
image 1	65,2	54,0	287,9	215,7	12,2	2,0	102,9	84,8
image 2	69,9	31,0	9234,7	3440,5	276,8	72,0	271,4	1601,1
image 3	713,1	288,0	136,3	65,9	321,3	3,2	77,3	47,5
image 4	103,6	86,8	160,2	158,0	5771,9	4168,6	8861,2	1776,9
Two images from same experiment included in training:
image 1	33,5	181,0	61,9	100,1	101,7	113,8	115,9	155,1
image 2	16,3	48,9	27,7	20,2	59,9	20,2	84,2	63,9

Discussion

We developed “DeepCLEM”, a fully automated CLEM registration workflow implemented in Fiji¹⁰ based on prediction of the chromatin stain from EM images using CNNs. Our registration workflow can easily be included in existing CLEM routines or adapted for imaging methods other than srAT where corresponding 2D slices need to be registered. If direct prediction of one modality from the other does not work, an alternative is to predict a common representation of both modalities, as described in Ref 15. While we found that "DeepCLEM" performs well under various conditions, it has some limitations: using chromatin staining for correlation requires the presence of at least three heterochromatin patches in the field of view. This limitation could be overcome by using e.g. propidium iodide to label the overall structure of the tissue. Widefield microscopy could be used where SIM is not available, but alignment quality is bounded by the lower-resolution channel.

The popular CLEM registration tool eC-CLEM⁵ has an “autofinder” function that detects corresponding features using spot finding or segmented regions. We did not perform a direct comparison, but results should be similar if suitable spots are found. If not, then image-to-image translation with DeepCLEM followed by point-based registration in eC-CLEM could be a promising alternative.

Data availability

Source code, pretrained networks and example data as well as documentation are available online at:

https://github.com/CIA-CCTB/Deep_CLEM.

Software availability

Source code available from: https://github.com/CIA-CCTB/Deep_CLEM.

Archived source code at time of publication: https://doi.org/10.5281/zenodo.4095247¹⁶

License: MIT License.

Acknowledgements

This publication was supported by COST Action NEUBIAS (CA15124), funded by COST (European Cooperation in Science and Technology). The JEOL JSM-7500F field emission scanning electron microscope (SEM) was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 218894895 (INST 93/761-1 FUGG) and the structured illumination microscope (SIM) was funded by DFG - 261184502 (INST 93/823-1 FUGG). VP and CS were funded by DFG within RTG 2581 SphingoINF (project 6).

Faculty Opinions recommended

References

1. Markert SM, Britz S, Proppert S, et al.: Filling the gap: adding super-resolution to array tomography for correlated ultrastructural and molecular identification of electrical synapses at the C. elegans connectome. Neurophotonics. 2016; 3(4): 041802. PubMed Abstract | Publisher Full Text | Free Full Text
2. Markert SM, Bauer V, Muenz TS, et al.: 3D subcellular localization with superresolution array tomography on ultrathin sections of various species. Methods Cell Biol. 2017; 140: 21–47. PubMed Abstract | Publisher Full Text
3. García-Fernández E, Koch G, Wagner RM, et al.: Membrane Microdomain Disassembly Inhibits MRSA Antibiotic Resistance. Cell. 2017; 171(6): 1354–1367.e20. PubMed Abstract | Publisher Full Text | Free Full Text
4. Cardona A, Saalfeld S, Schindelin J, et al.: TrakEM2 Software for Neural Circuit Reconstruction. PLoS One. 2012; 7(6): e38011. PubMed Abstract | Publisher Full Text | Free Full Text
5. Paul-Gilloteaux P, Heiligenstein X, Belle M, et al.: eC-CLEM: flexible multidimensional registration software for correlative microscopies. Nat Methods. 2017; 14(2): 102–103. PubMed Abstract | Publisher Full Text
6. von Chamier L, Laine RF, Henriques R: Artificial intelligence for microscopy: what you should know. Biochem Soc Trans. 2019; 47(4): 1029–1040. PubMed Abstract | Publisher Full Text
7. Belthangady C, Royer LA: Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat Methods. 2019; 16(12): 1215–1225. PubMed Abstract | Publisher Full Text
8. Christiansen EM, Yang SJ, Ando DM, et al.: In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images. Cell. 2018; 173(3): 792–803.e19. PubMed Abstract | Publisher Full Text | Free Full Text
9. Guo SM, Yeh LH, Folkesson J, et al.: Revealing architectural order with quantitative label-free imaging and deep learning. eLife. 2020; 9: e55502. PubMed Abstract | Publisher Full Text | Free Full Text
10. Schindelin J, Arganda-Carreras I, Frise E, et al.: Fiji: an open-source platform for biological-image analysis. Nat Methods. 2012; 9(7): 676–682. PubMed Abstract | Publisher Full Text | Free Full Text
11. Uçeyler N, Kafke W, Riediger N, et al.: Elevated proinflammatory cytokine expression in affected skin in small fiber neuropathy. Neurology. 2010; 74(22): 1806–1813. PubMed Abstract | Publisher Full Text
12. Weigert M, Schmidt U, Boothe T, et al.: Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat Methods. 2018; 15(12): 1090–1097. PubMed Abstract | Publisher Full Text
13. Virtanen P, Gommers R, Oliphant TE, et al.: SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods. 2020; 17(3): 261–272. PubMed Abstract | Publisher Full Text | Free Full Text
14. Abadi M, Agarwal A, Barham P, et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. arXiv: 160304467 [cs]. 2016. Publisher Full Text
15. Pielawski N, Wetzer E, Öfverstedt J, et al.: CoMIR: Contrastive Multimodal Image Representation for Registration. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., 2020; 33: 18433–18444. Publisher Full Text
16. Seifert R: CIA-CCTB/Deep_CLEM: First release of DeepCLEM (Version v1.0). Zenodo. 2020. http://www.doi.org/10.5281/zenodo.4095247

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 27 Oct 2020

Author details Author details

Rick Seifert
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Software, Validation, Writing – Original Draft Preparation, Writing – Review & Editing

Sebastian M. Markert
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Sebastian Britz
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Veronika Perschin
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Christoph Erbacher
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Christian Stigloher
Roles: Conceptualization, Data Curation, Funding Acquisition, Investigation, Methodology, Resources, Supervision, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

SMM was supported by a PhD grant from the Studienstiftung des Deutschen Volkes (German Academic Scholarship Foundation). VP and CS are funded by Deutsche Forschungsgemeinschaft GRK2581 SPHINGOINF and CS by STI 700/1-1. Computational work was performed using the High-Performance Computing Cloud of Würzburg University (DFG project 327497565).This publication was supported by COST Action NEUBIAS (CA15124), funded by COST (European Cooperation in Science and Technology). The JEOL JSM-7500F field emission scanning
electron microscope (SEM) was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 218894895 (INST 93/761-1 FUGG) and the structured illumination
microscope (SIM) was funded by DFG - 261184502 (INST 93/823-1 FUGG). VP and CS were funded by DFG within RTG 2581 SphingoINF (project 6).

Article Versions (3)

version 3

Revised

Published: 28 Dec 2023, 9:1275

https://doi.org/10.12688/f1000research.27158.3

version 2

Revised

Published: 02 Aug 2022, 9:1275

https://doi.org/10.12688/f1000research.27158.2

version 1

Published: 27 Oct 2020, 9:1275

https://doi.org/10.12688/f1000research.27158.1

© 2022 Seifert R et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Seifert R, Markert SM, Britz S et al. DeepCLEM: automated registration for correlative light and electron microscopy using deep learning [version 2; peer review: 1 approved, 2 approved with reservations]. F1000Research 2022, 9:1275 (https://doi.org/10.12688/f1000research.27158.2)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 2

VERSION 2

PUBLISHED 02 Aug 2022

Revised

Views

Reviewer Report 29 Jun 2023

Christopher Schmied, AG Han Sun, Leibniz-Forschungsinstitut für Molekulare Pharmakologie, Berlin, Berlin, Germany

Approved with Reservations

https://doi.org/10.5256/f1000research.136822.r179427

The authors present an image registration workflow to register light microscopy (LM) images into electron microscopy (EM) images for correlative light-electron microscopy (CLEM). To solve the multimodal registration problem, DeepCLEM predicts a LM image from the EM image using ProjectionCARE. The predicted LM image is then used to compute the necessary transformation of the LM image to the EM image. Apart from the pre-trained model, which is limited to data it has been trained on, the authors also describe how to train the model on their own data.

Overall, a very nice approach that has been well implemented.

In my review I verified if the provided software package and its documentation allows the successful reproduction of the described workflow. I also assessed if the provided article sufficiently describes and discusses the proposed approach (i.e. scientific context and limitations).

Application of the pre-trained model:

I was able to easily install the software as per documentation. The provided test images allow to easily verify the correct operation of the Fiji workflow. The output data is clearly and sufficiently described. Positive is that the transformation file is provided that allows to apply the transformation on other data.

I found some points problematic in the documentation:
"The EM image should look like the testing EM.png." This description is too unspecific for novice users. Important would be here to emphasize that this is in context to the application of the pre-trained model on new data.

Retraining:

Great that the conda .yml is provided for easy installation of the dependencies. As a warning, I experienced that the creation of the conda environment took a long time on my machine. But could be specific to my setup. The possibilities of my review here are also limited since I do not have the possibility to generate my own training data from scratch and I can only test the provided data. Within these constraints, I was able to test the notebooks for their correct operation.

Article:

The article is overall well written and explains the context of the work. The authors also test and discuss the limitations of the approach. Further, providing avenues to overcome problems and limitations.

As a side note, not relevant to the review, in terms of the registration field in the biology focused sciences, I am always a bit surprised that software solutions such as elastix (https://elastix.lumc.nl/) are not more widely used and discussed. I would have expected that the solutions of multimodal image registration developed in the medical field would be at least somewhat relevant to CLEM. The article seems to also discuss another established registration tool (eC-CLEM) so there is sufficient other context for this publication.

The methods section of the article is unfortunately a bit sparse. The actual network architecture is only mentioned in the results. No further detail is provided. It is fine if the default values of the out-of-the-box network are used. However, this should be clearly stated in the methods. I would also wish a better description of how the training has been performed. Including how long the training took and on which hardware this was performed.

I really enjoyed that the authors provide all necessary code and example images on their GitHub repo. The only tiny thing missing is to clearly state their preferred way of users getting in touch for support. I would suggest clearly stating this in the software availability section.

I endorse the article under the following conditions:

A more extensive methods section. Describing clearer what the architecture was and any important or changed hyperparameters. Also, the hardware and the training time should be specified.
Improve documentation from the perspective of a novice user.
State how users should get in touch for support.

Is the rationale for developing the new software tool clearly explained?

Yes
Is the description of the software tool technically sound?

Yes
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Data science, bioimage analysis, cell biology.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 13 Apr 2024

Philip Kollmannsberger, Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany

13 Apr 2024

Author Response

Authors: Thank you for the detailed evaluation and feedback – we greatly appreciate your time and effort!

Reviewer: I found some points problematic in the documentation:
"The EM image should ... Continue reading Authors: Thank you for the detailed evaluation and feedback – we greatly appreciate your time and effort!

Reviewer: I found some points problematic in the documentation:
"The EM image should look like the testing EM.png." This description is too unspecific for novice users. Important would be here to emphasize that this is in context to the application of the pre-trained model on new data.

Authors: We now improved the documentation and tried to make it more accessible for novice users. The instructions on the requirements for the images to be aligned are now more detailed, and a better explanation on when to train a custom model was included.

Reviewer: Great that the conda .yml is provided for easy installation of the dependencies. As a warning, I experienced that the creation of the conda environment took a long time on my machine. But could be specific to my setup. The possibilities of my review here are also limited since I do not have the possibility to generate my own training data from scratch and I can only test the provided data. Within these constraints, I was able to test the notebooks for their correct operation.

Authors: To improve the installation of the dependencies for network training, we optimized the .yml file for the conda environment. Furthermore, we included mamba as package manager in the documentation to speed up solving the environment. The modified environment file was tested under Windows, Mac M1 and Linux.

Reviewer: The methods section of the article is unfortunately a bit sparse. The actual network architecture is only mentioned in the results. No further detail is provided. It is fine if the default values of the out-of-the-box network are used. However, this should be clearly stated in the methods. I would also wish a better description of how the training has been performed. Including how long the training took and on which hardware this was performed.

Authors: We updated the methods section and added a new paragraph to include information about the training parameters, the hardware that was used and how long the training took. The new paragraph at the end of the “Implementation” subsection reads:

“We trained the ProjectionCARE 2D UNet architecture from CSBDeep12 with 3 layers and a batch size of 8 for 150 epochs, using the default parameters of CSBDeep unless stated otherwise. Training on a GPU took between one (RTX 2080 Ti, cuda 11.3, Windows 10) and three hours (Tesla P100, cuda 11.7, Ubuntu 20.04) and around 10 hours on a CPU (Intel i7-8700K at 3.70 GHz, Windows 10). The Jupyter notebook for training on custom datasets as well as documentation how to use it are available online (see software availability statement).”

Reviewer: I really enjoyed that the authors provide all necessary code and example images on their GitHub repo. The only tiny thing missing is to clearly state their preferred way of users getting in touch for support. I would suggest clearly stating this in the software availability section.

Authors: This information was indeed missing, we now include contact information for support both in the manuscript as well as in the documentation. The software availability section now contains information on how to contact the authors for support and questions. We also added a more detailed section on how to get in touch to the beginning of the documentation at the repository.
Authors: Thank you for the detailed evaluation and feedback – we greatly appreciate your time and effort!

Reviewer: I found some points problematic in the documentation:
"The EM image should look like the testing EM.png." This description is too unspecific for novice users. Important would be here to emphasize that this is in context to the application of the pre-trained model on new data.

Authors: We now improved the documentation and tried to make it more accessible for novice users. The instructions on the requirements for the images to be aligned are now more detailed, and a better explanation on when to train a custom model was included.

Reviewer: Great that the conda .yml is provided for easy installation of the dependencies. As a warning, I experienced that the creation of the conda environment took a long time on my machine. But could be specific to my setup. The possibilities of my review here are also limited since I do not have the possibility to generate my own training data from scratch and I can only test the provided data. Within these constraints, I was able to test the notebooks for their correct operation.

Authors: To improve the installation of the dependencies for network training, we optimized the .yml file for the conda environment. Furthermore, we included mamba as package manager in the documentation to speed up solving the environment. The modified environment file was tested under Windows, Mac M1 and Linux.

Reviewer: The methods section of the article is unfortunately a bit sparse. The actual network architecture is only mentioned in the results. No further detail is provided. It is fine if the default values of the out-of-the-box network are used. However, this should be clearly stated in the methods. I would also wish a better description of how the training has been performed. Including how long the training took and on which hardware this was performed.

Authors: We updated the methods section and added a new paragraph to include information about the training parameters, the hardware that was used and how long the training took. The new paragraph at the end of the “Implementation” subsection reads:

“We trained the ProjectionCARE 2D UNet architecture from CSBDeep12 with 3 layers and a batch size of 8 for 150 epochs, using the default parameters of CSBDeep unless stated otherwise. Training on a GPU took between one (RTX 2080 Ti, cuda 11.3, Windows 10) and three hours (Tesla P100, cuda 11.7, Ubuntu 20.04) and around 10 hours on a CPU (Intel i7-8700K at 3.70 GHz, Windows 10). The Jupyter notebook for training on custom datasets as well as documentation how to use it are available online (see software availability statement).”

Reviewer: I really enjoyed that the authors provide all necessary code and example images on their GitHub repo. The only tiny thing missing is to clearly state their preferred way of users getting in touch for support. I would suggest clearly stating this in the software availability section.

Authors: This information was indeed missing, we now include contact information for support both in the manuscript as well as in the documentation. The software availability section now contains information on how to contact the authors for support and questions. We also added a more detailed section on how to get in touch to the beginning of the documentation at the repository.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 13 Apr 2024

Philip Kollmannsberger, Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany

13 Apr 2024

Author Response

Authors: Thank you for the detailed evaluation and feedback – we greatly appreciate your time and effort!

Reviewer: I found some points problematic in the documentation:
"The EM image should ... Continue reading Authors: Thank you for the detailed evaluation and feedback – we greatly appreciate your time and effort!

Reviewer: I found some points problematic in the documentation:
"The EM image should look like the testing EM.png." This description is too unspecific for novice users. Important would be here to emphasize that this is in context to the application of the pre-trained model on new data.

Authors: We now improved the documentation and tried to make it more accessible for novice users. The instructions on the requirements for the images to be aligned are now more detailed, and a better explanation on when to train a custom model was included.

Reviewer: Great that the conda .yml is provided for easy installation of the dependencies. As a warning, I experienced that the creation of the conda environment took a long time on my machine. But could be specific to my setup. The possibilities of my review here are also limited since I do not have the possibility to generate my own training data from scratch and I can only test the provided data. Within these constraints, I was able to test the notebooks for their correct operation.

Authors: To improve the installation of the dependencies for network training, we optimized the .yml file for the conda environment. Furthermore, we included mamba as package manager in the documentation to speed up solving the environment. The modified environment file was tested under Windows, Mac M1 and Linux.

Reviewer: The methods section of the article is unfortunately a bit sparse. The actual network architecture is only mentioned in the results. No further detail is provided. It is fine if the default values of the out-of-the-box network are used. However, this should be clearly stated in the methods. I would also wish a better description of how the training has been performed. Including how long the training took and on which hardware this was performed.

Authors: We updated the methods section and added a new paragraph to include information about the training parameters, the hardware that was used and how long the training took. The new paragraph at the end of the “Implementation” subsection reads:

“We trained the ProjectionCARE 2D UNet architecture from CSBDeep12 with 3 layers and a batch size of 8 for 150 epochs, using the default parameters of CSBDeep unless stated otherwise. Training on a GPU took between one (RTX 2080 Ti, cuda 11.3, Windows 10) and three hours (Tesla P100, cuda 11.7, Ubuntu 20.04) and around 10 hours on a CPU (Intel i7-8700K at 3.70 GHz, Windows 10). The Jupyter notebook for training on custom datasets as well as documentation how to use it are available online (see software availability statement).”

Reviewer: I really enjoyed that the authors provide all necessary code and example images on their GitHub repo. The only tiny thing missing is to clearly state their preferred way of users getting in touch for support. I would suggest clearly stating this in the software availability section.

Authors: This information was indeed missing, we now include contact information for support both in the manuscript as well as in the documentation. The software availability section now contains information on how to contact the authors for support and questions. We also added a more detailed section on how to get in touch to the beginning of the documentation at the repository.
Authors: Thank you for the detailed evaluation and feedback – we greatly appreciate your time and effort!

Reviewer: I found some points problematic in the documentation:
"The EM image should look like the testing EM.png." This description is too unspecific for novice users. Important would be here to emphasize that this is in context to the application of the pre-trained model on new data.

Authors: We now improved the documentation and tried to make it more accessible for novice users. The instructions on the requirements for the images to be aligned are now more detailed, and a better explanation on when to train a custom model was included.

Reviewer: Great that the conda .yml is provided for easy installation of the dependencies. As a warning, I experienced that the creation of the conda environment took a long time on my machine. But could be specific to my setup. The possibilities of my review here are also limited since I do not have the possibility to generate my own training data from scratch and I can only test the provided data. Within these constraints, I was able to test the notebooks for their correct operation.

Authors: To improve the installation of the dependencies for network training, we optimized the .yml file for the conda environment. Furthermore, we included mamba as package manager in the documentation to speed up solving the environment. The modified environment file was tested under Windows, Mac M1 and Linux.

Reviewer: The methods section of the article is unfortunately a bit sparse. The actual network architecture is only mentioned in the results. No further detail is provided. It is fine if the default values of the out-of-the-box network are used. However, this should be clearly stated in the methods. I would also wish a better description of how the training has been performed. Including how long the training took and on which hardware this was performed.

Authors: We updated the methods section and added a new paragraph to include information about the training parameters, the hardware that was used and how long the training took. The new paragraph at the end of the “Implementation” subsection reads:

“We trained the ProjectionCARE 2D UNet architecture from CSBDeep12 with 3 layers and a batch size of 8 for 150 epochs, using the default parameters of CSBDeep unless stated otherwise. Training on a GPU took between one (RTX 2080 Ti, cuda 11.3, Windows 10) and three hours (Tesla P100, cuda 11.7, Ubuntu 20.04) and around 10 hours on a CPU (Intel i7-8700K at 3.70 GHz, Windows 10). The Jupyter notebook for training on custom datasets as well as documentation how to use it are available online (see software availability statement).”

Reviewer: I really enjoyed that the authors provide all necessary code and example images on their GitHub repo. The only tiny thing missing is to clearly state their preferred way of users getting in touch for support. I would suggest clearly stating this in the software availability section.

Authors: This information was indeed missing, we now include contact information for support both in the manuscript as well as in the documentation. The software availability section now contains information on how to contact the authors for support and questions. We also added a more detailed section on how to get in touch to the beginning of the documentation at the repository.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 30 Aug 2022

Martin L. Jones, Electron Microscopy Science Technology Platform, The Francis Crick Institute, London, UK

Approved

https://doi.org/10.5256/f1000research.136822.r146290

No further comments, the authors have ... Continue reading

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 27 Oct 2020

Views

Reviewer Report 08 Feb 2021

Reinhard Rachel, Centre for Electron Microscopy, Faculty of Biology and Preclinical Medicine, University of Regensburg, Regensburg, Germany

Approved with Reservations

https://doi.org/10.5256/f1000research.30002.r77401

The manuscript by Seifert et al. (Univ. Würzburg, Germany) provides a convenient software tool (a macro name DeepCLEM) for linking fluorescent light microscopy images (here: structured illumination microscopy) to the information seen in electron micrographs (in this particular case: FE-SEM), obtained from serial sections on slides.
In a first glance, this short paper describes the development and application of a flexible and easy to handle software tool to a program which is open to many microscopy researchers, ImageJ / Fiji, in form of a macro, which can easily be added to any Fiji installation. Thus, it appears that this tool can be used by many groups aiming to analyze biological samples using correleative light and electron microscopy (CLEM).
A few questions arise when reading the manuscript - which do not modify the overall positive impression of this study.

The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere?
What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed? - the authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow?
Histogram equalization of EM images - was this done with routines implemented in Fiji?
Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM? or at least comment on this?

With comments on my points listed above, and according improvements in the manuscript, this can be indexed - thus, approved with reservations.

Is the rationale for developing the new software tool clearly explained?

No
Is the description of the software tool technically sound?

Partly
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: TEM, STEM, tomography, CLEM; microbiology, cell biology

CITE

Report a concern

Author Response 16 Aug 2022

Philip Kollmannsberger, Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany

16 Aug 2022

Author Response

R: The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, ... Continue reading R: The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere?

A: If the structures used for alignment (e.g., chromatin) can be resolved and the WF images can be manually aligned, it is also possible to obtain a prediction and do automated alignment, although the achievable quality of the registration is obviously limited by the resolution of the fluorescence channel. We now added this in the discussion of the scope and limitations of the tool.

R: What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed?

A: Using a trained network for prediction and alignment can be done fully automatically in Fiji, only the paths of the images and network must be provided in the user dialog. For retraining, some knowledge (e.g., setting up a python environment) is necessary. We now added a more detailed description of the training workflow. Currently, the training process should probably be performed by some more experienced person while using the trained network does not require any special knowledge.

R: The authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow?

A: It may be possible to use the provided network trained on our data on images from other labs directly, however in most cases re-training is required. We now describe the required workflow in more detail and perform a quantitative comparison of registration quality where images from an experiment were either included in the training set or not.

R: Histogram equalization of EM images - was this done with routines implemented in Fiji?

A: We used the standard histogram equalization implemented in Fiji. We added this detail to the method description.

R: Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM? or at least comment on this?

A: We now comment on ec-CLEM and its “autofinder” functionality which is closest to our tool in terms of functionality. While ec-CLEM autofinder uses spot finding or centers of segmented regions as points of interest to perform registration on, we predict the image of the other modality and then use correlation-based alignment.
R: The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere?

A: If the structures used for alignment (e.g., chromatin) can be resolved and the WF images can be manually aligned, it is also possible to obtain a prediction and do automated alignment, although the achievable quality of the registration is obviously limited by the resolution of the fluorescence channel. We now added this in the discussion of the scope and limitations of the tool.

R: What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed?

A: Using a trained network for prediction and alignment can be done fully automatically in Fiji, only the paths of the images and network must be provided in the user dialog. For retraining, some knowledge (e.g., setting up a python environment) is necessary. We now added a more detailed description of the training workflow. Currently, the training process should probably be performed by some more experienced person while using the trained network does not require any special knowledge.

R: The authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow?

A: It may be possible to use the provided network trained on our data on images from other labs directly, however in most cases re-training is required. We now describe the required workflow in more detail and perform a quantitative comparison of registration quality where images from an experiment were either included in the training set or not.

R: Histogram equalization of EM images - was this done with routines implemented in Fiji?

A: We used the standard histogram equalization implemented in Fiji. We added this detail to the method description.

R: Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM? or at least comment on this?

A: We now comment on ec-CLEM and its “autofinder” functionality which is closest to our tool in terms of functionality. While ec-CLEM autofinder uses spot finding or centers of segmented regions as points of interest to perform registration on, we predict the image of the other modality and then use correlation-based alignment.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 16 Aug 2022

Philip Kollmannsberger, Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany

16 Aug 2022

Author Response

R: The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, ... Continue reading R: The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere?

A: If the structures used for alignment (e.g., chromatin) can be resolved and the WF images can be manually aligned, it is also possible to obtain a prediction and do automated alignment, although the achievable quality of the registration is obviously limited by the resolution of the fluorescence channel. We now added this in the discussion of the scope and limitations of the tool.

R: What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed?

A: Using a trained network for prediction and alignment can be done fully automatically in Fiji, only the paths of the images and network must be provided in the user dialog. For retraining, some knowledge (e.g., setting up a python environment) is necessary. We now added a more detailed description of the training workflow. Currently, the training process should probably be performed by some more experienced person while using the trained network does not require any special knowledge.

R: The authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow?

A: It may be possible to use the provided network trained on our data on images from other labs directly, however in most cases re-training is required. We now describe the required workflow in more detail and perform a quantitative comparison of registration quality where images from an experiment were either included in the training set or not.

R: Histogram equalization of EM images - was this done with routines implemented in Fiji?

A: We used the standard histogram equalization implemented in Fiji. We added this detail to the method description.

R: Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM? or at least comment on this?

A: We now comment on ec-CLEM and its “autofinder” functionality which is closest to our tool in terms of functionality. While ec-CLEM autofinder uses spot finding or centers of segmented regions as points of interest to perform registration on, we predict the image of the other modality and then use correlation-based alignment.
R: The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere?

A: If the structures used for alignment (e.g., chromatin) can be resolved and the WF images can be manually aligned, it is also possible to obtain a prediction and do automated alignment, although the achievable quality of the registration is obviously limited by the resolution of the fluorescence channel. We now added this in the discussion of the scope and limitations of the tool.

R: What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed?

A: Using a trained network for prediction and alignment can be done fully automatically in Fiji, only the paths of the images and network must be provided in the user dialog. For retraining, some knowledge (e.g., setting up a python environment) is necessary. We now added a more detailed description of the training workflow. Currently, the training process should probably be performed by some more experienced person while using the trained network does not require any special knowledge.

R: The authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow?

A: It may be possible to use the provided network trained on our data on images from other labs directly, however in most cases re-training is required. We now describe the required workflow in more detail and perform a quantitative comparison of registration quality where images from an experiment were either included in the training set or not.

R: Histogram equalization of EM images - was this done with routines implemented in Fiji?

A: We used the standard histogram equalization implemented in Fiji. We added this detail to the method description.

R: Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM? or at least comment on this?

A: We now comment on ec-CLEM and its “autofinder” functionality which is closest to our tool in terms of functionality. While ec-CLEM autofinder uses spot finding or centers of segmented regions as points of interest to perform registration on, we predict the image of the other modality and then use correlation-based alignment.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 04 Feb 2021

Martin L. Jones, Electron Microscopy Science Technology Platform, The Francis Crick Institute, London, UK

Approved with Reservations

https://doi.org/10.5256/f1000research.30002.r77801

The authors present an automated method, deployed as a Fiji plugin, for registering 2D CLEM data. This is demonstrated on a "super resolution Array Tomography" (srAT) dataset, building on their previous method development work, where the sample is sliced into 100nm sections prior to imaging in SIM and SEM. The DeepCLEM plugin generates synthetic fluorescence data from the EM images that are aligned to the real fluorescence signal to perform the registration.

This type of effort is crucial in harnessing the huge potential of correlative multimodal imaging as cross-modal registration is a significant bottleneck in many cases. The method of imaging both LM and EM on the processed and cut sections is perhaps not as widespread as other methods, such as, e.g. whole cell confocal imaging followed by resin-embedding, followed by EM imaging. The method used in this paper avoids the familiar CLEM registration problems of non-linear warping induced by the EM sample preparation and mismatched axial resolutions. This constrains the registration task to a "similarity" type transform rather than affine or non-rigid as is required in some other CLEM registration tasks.

The open source plugin is easy to install and runs as described on the test data provided.

A few points that I think should be addressed:

In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.
The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo?
Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.
I believe the preferred capitalisation is "Fiji" and not "FIJI"
ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.
For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.
In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.
If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists.
The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.
An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial!
The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well.
Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract.
The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful

Overall, the tool itself works as described on the data type provided, but I think there is a risk that users might be tempted to stretch beyond the current scope of the tool, so this scope should be made more clear in the text.

Is the rationale for developing the new software tool clearly explained?

Yes
Is the description of the software tool technically sound?

Partly
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Image Analysis, CLEM, Machine Learning, Electron microscopy

CITE

Report a concern

Author Response 16 Aug 2022

Philip Kollmannsberger, Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany

16 Aug 2022

Author Response

R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment ... Continue reading R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.

A: We removed the mention of a specific stain in the abstract to reflect the general applicability of the method.

R: The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo?

A: The code is a Jython script with graphical user interface to be run within Fiji. From the user's perspective, it is indeed a plugin. We changed “macro” to “plugin”, as suggested.

R: Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.

A: The trained network can be applied to different datasets if they have similar contrast, usually from the same microscope or sample type. Otherwise, (re-)training of the network is necessary. We now make this clearer in the manuscript and include a description of the training workflow.

R: I believe the preferred capitalisation is "Fiji" and not "FIJI"

A: Thank you, we corrected this.

R: ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.

A: We now refer to ec-CLEM autofinder which is similar in terms of functionality. It is based on finding corresponding features in both modalities, e.g., using wavelet-based spot detection or the centers of segmented regions, and then performing registration of these automatically detected landmarks. We did not perform a direct quantitative comparison. If suitable corresponding spots can be found (e.g. beads), the performance on the same images using the same registration algorithm should be comparable. In cases where spot finding or similar methods in ec-CLEM do not perform well because the images are too dissimilar or if there are no fiducial markers, then transferring the EM image to fluorescence using DeepCLEM could be used to generate input data for point-based registration in ec-CLEM.

R: For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.

A: We agree that these methods are more reproducible and added an additional note to the paragraph on manual registration to encourage users to use such reproducible methods. Since we already had created the training data, we did, however, not repeat the alignment. For assessing the performance (see below), we instead artificially displaced the two channels - in this case, the steps and transforms are also known exactly.

R: In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.

A: We now include more detail about the training workflow in the implementation section and compared the performance for different amounts of training images.

R: If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists.

A: We extended both the description of the training parameters as well as the documentation of the entire workflow, including training in the Jupyter notebook.

R: The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.

A: The text now explicitly mentions the use of the "Register Virtual Stack Slices" plugin and the "similarity" transform in our work, but in principle, our workflow should also work with other means of correlation-based registration.

R: An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial!

A: We now performed quantification of the quality of the registration using the manually aligned ground truth data as a reference where the best solution is already known. We observed that in most cases, the registration either works and has a very small error or is completely off by several 100 nm, which should be easy to spot. If a few images from an experiment are manually aligned and included in the training set, the registration of the remaining images improves significantly (Table 1).

Instead of providing a confidence metric, which would indeed be non-trivial, we recommend visual inspection using the overlay. Possible ways to implement a confidence metric would be to output a confidence level for the neural network prediction, and to check the quality of the alignment using a correlation metric between fluorescent and predicted image.

R: The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well.

A: We added more detail regarding the scope and limitations of this tool and its current requirements to the abstract, introduction, and discussion section, including the need for at least three heterochromatin patches if chromatin staining is used for registration.

R: Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract. The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful

A: The way it is currently configured, DeepCLEM can in principle be used to register any CLEM data where corresponding 2D slices are available in both modalities, whether they were acquired on the same cut sections (srAT) or prior and post sectioning. The examples we provide however are restricted to srAT. We now state this in the text.
R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.

A: We removed the mention of a specific stain in the abstract to reflect the general applicability of the method.

R: The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo?

A: The code is a Jython script with graphical user interface to be run within Fiji. From the user's perspective, it is indeed a plugin. We changed “macro” to “plugin”, as suggested.

R: Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.

A: The trained network can be applied to different datasets if they have similar contrast, usually from the same microscope or sample type. Otherwise, (re-)training of the network is necessary. We now make this clearer in the manuscript and include a description of the training workflow.

R: I believe the preferred capitalisation is "Fiji" and not "FIJI"

A: Thank you, we corrected this.

R: ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.

A: We now refer to ec-CLEM autofinder which is similar in terms of functionality. It is based on finding corresponding features in both modalities, e.g., using wavelet-based spot detection or the centers of segmented regions, and then performing registration of these automatically detected landmarks. We did not perform a direct quantitative comparison. If suitable corresponding spots can be found (e.g. beads), the performance on the same images using the same registration algorithm should be comparable. In cases where spot finding or similar methods in ec-CLEM do not perform well because the images are too dissimilar or if there are no fiducial markers, then transferring the EM image to fluorescence using DeepCLEM could be used to generate input data for point-based registration in ec-CLEM.

R: For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.

A: We agree that these methods are more reproducible and added an additional note to the paragraph on manual registration to encourage users to use such reproducible methods. Since we already had created the training data, we did, however, not repeat the alignment. For assessing the performance (see below), we instead artificially displaced the two channels - in this case, the steps and transforms are also known exactly.

R: In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.

A: We now include more detail about the training workflow in the implementation section and compared the performance for different amounts of training images.

R: If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists.

A: We extended both the description of the training parameters as well as the documentation of the entire workflow, including training in the Jupyter notebook.

R: The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.

A: The text now explicitly mentions the use of the "Register Virtual Stack Slices" plugin and the "similarity" transform in our work, but in principle, our workflow should also work with other means of correlation-based registration.

R: An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial!

A: We now performed quantification of the quality of the registration using the manually aligned ground truth data as a reference where the best solution is already known. We observed that in most cases, the registration either works and has a very small error or is completely off by several 100 nm, which should be easy to spot. If a few images from an experiment are manually aligned and included in the training set, the registration of the remaining images improves significantly (Table 1).

Instead of providing a confidence metric, which would indeed be non-trivial, we recommend visual inspection using the overlay. Possible ways to implement a confidence metric would be to output a confidence level for the neural network prediction, and to check the quality of the alignment using a correlation metric between fluorescent and predicted image.

R: The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well.

A: We added more detail regarding the scope and limitations of this tool and its current requirements to the abstract, introduction, and discussion section, including the need for at least three heterochromatin patches if chromatin staining is used for registration.

R: Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract. The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful

A: The way it is currently configured, DeepCLEM can in principle be used to register any CLEM data where corresponding 2D slices are available in both modalities, whether they were acquired on the same cut sections (srAT) or prior and post sectioning. The examples we provide however are restricted to srAT. We now state this in the text.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 16 Aug 2022

Philip Kollmannsberger, Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany

16 Aug 2022

Author Response

R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment ... Continue reading R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.

A: We removed the mention of a specific stain in the abstract to reflect the general applicability of the method.

R: The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo?

A: The code is a Jython script with graphical user interface to be run within Fiji. From the user's perspective, it is indeed a plugin. We changed “macro” to “plugin”, as suggested.

R: Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.

A: The trained network can be applied to different datasets if they have similar contrast, usually from the same microscope or sample type. Otherwise, (re-)training of the network is necessary. We now make this clearer in the manuscript and include a description of the training workflow.

R: I believe the preferred capitalisation is "Fiji" and not "FIJI"

A: Thank you, we corrected this.

R: ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.

A: We now refer to ec-CLEM autofinder which is similar in terms of functionality. It is based on finding corresponding features in both modalities, e.g., using wavelet-based spot detection or the centers of segmented regions, and then performing registration of these automatically detected landmarks. We did not perform a direct quantitative comparison. If suitable corresponding spots can be found (e.g. beads), the performance on the same images using the same registration algorithm should be comparable. In cases where spot finding or similar methods in ec-CLEM do not perform well because the images are too dissimilar or if there are no fiducial markers, then transferring the EM image to fluorescence using DeepCLEM could be used to generate input data for point-based registration in ec-CLEM.

R: For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.

A: We agree that these methods are more reproducible and added an additional note to the paragraph on manual registration to encourage users to use such reproducible methods. Since we already had created the training data, we did, however, not repeat the alignment. For assessing the performance (see below), we instead artificially displaced the two channels - in this case, the steps and transforms are also known exactly.

R: In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.

A: We now include more detail about the training workflow in the implementation section and compared the performance for different amounts of training images.

R: If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists.

A: We extended both the description of the training parameters as well as the documentation of the entire workflow, including training in the Jupyter notebook.

R: The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.

A: The text now explicitly mentions the use of the "Register Virtual Stack Slices" plugin and the "similarity" transform in our work, but in principle, our workflow should also work with other means of correlation-based registration.

R: An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial!

A: We now performed quantification of the quality of the registration using the manually aligned ground truth data as a reference where the best solution is already known. We observed that in most cases, the registration either works and has a very small error or is completely off by several 100 nm, which should be easy to spot. If a few images from an experiment are manually aligned and included in the training set, the registration of the remaining images improves significantly (Table 1).

Instead of providing a confidence metric, which would indeed be non-trivial, we recommend visual inspection using the overlay. Possible ways to implement a confidence metric would be to output a confidence level for the neural network prediction, and to check the quality of the alignment using a correlation metric between fluorescent and predicted image.

R: The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well.

A: We added more detail regarding the scope and limitations of this tool and its current requirements to the abstract, introduction, and discussion section, including the need for at least three heterochromatin patches if chromatin staining is used for registration.

R: Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract. The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful

A: The way it is currently configured, DeepCLEM can in principle be used to register any CLEM data where corresponding 2D slices are available in both modalities, whether they were acquired on the same cut sections (srAT) or prior and post sectioning. The examples we provide however are restricted to srAT. We now state this in the text.
R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.

A: We removed the mention of a specific stain in the abstract to reflect the general applicability of the method.

R: The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo?

A: The code is a Jython script with graphical user interface to be run within Fiji. From the user's perspective, it is indeed a plugin. We changed “macro” to “plugin”, as suggested.

R: Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.

A: The trained network can be applied to different datasets if they have similar contrast, usually from the same microscope or sample type. Otherwise, (re-)training of the network is necessary. We now make this clearer in the manuscript and include a description of the training workflow.

R: I believe the preferred capitalisation is "Fiji" and not "FIJI"

A: Thank you, we corrected this.

R: ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.

A: We now refer to ec-CLEM autofinder which is similar in terms of functionality. It is based on finding corresponding features in both modalities, e.g., using wavelet-based spot detection or the centers of segmented regions, and then performing registration of these automatically detected landmarks. We did not perform a direct quantitative comparison. If suitable corresponding spots can be found (e.g. beads), the performance on the same images using the same registration algorithm should be comparable. In cases where spot finding or similar methods in ec-CLEM do not perform well because the images are too dissimilar or if there are no fiducial markers, then transferring the EM image to fluorescence using DeepCLEM could be used to generate input data for point-based registration in ec-CLEM.

R: For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.

A: We agree that these methods are more reproducible and added an additional note to the paragraph on manual registration to encourage users to use such reproducible methods. Since we already had created the training data, we did, however, not repeat the alignment. For assessing the performance (see below), we instead artificially displaced the two channels - in this case, the steps and transforms are also known exactly.

R: In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.

A: We now include more detail about the training workflow in the implementation section and compared the performance for different amounts of training images.

R: If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists.

A: We extended both the description of the training parameters as well as the documentation of the entire workflow, including training in the Jupyter notebook.

R: The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.

A: The text now explicitly mentions the use of the "Register Virtual Stack Slices" plugin and the "similarity" transform in our work, but in principle, our workflow should also work with other means of correlation-based registration.

R: An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial!

A: We now performed quantification of the quality of the registration using the manually aligned ground truth data as a reference where the best solution is already known. We observed that in most cases, the registration either works and has a very small error or is completely off by several 100 nm, which should be easy to spot. If a few images from an experiment are manually aligned and included in the training set, the registration of the remaining images improves significantly (Table 1).

Instead of providing a confidence metric, which would indeed be non-trivial, we recommend visual inspection using the overlay. Possible ways to implement a confidence metric would be to output a confidence level for the neural network prediction, and to check the quality of the alignment using a correlation metric between fluorescent and predicted image.

R: The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well.

A: We added more detail regarding the scope and limitations of this tool and its current requirements to the abstract, introduction, and discussion section, including the need for at least three heterochromatin patches if chromatin staining is used for registration.

R: Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract. The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful

A: The way it is currently configured, DeepCLEM can in principle be used to register any CLEM data where corresponding 2D slices are available in both modalities, whether they were acquired on the same cut sections (srAT) or prior and post sectioning. The examples we provide however are restricted to srAT. We now state this in the text.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 27 Oct 2020

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3
Version 3 (revision) 28 Dec 23	read
Version 2 (revision) 02 Aug 22	read		read
Version 1 27 Oct 20	read	read

Martin L. Jones, The Francis Crick Institute, London, UK
Reinhard Rachel, University of Regensburg, Regensburg, Germany
Christopher Schmied, Leibniz-Forschungsinstitut für Molekulare Pharmakologie, Berlin, Germany

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

4 Views

25 Jan 2024 | for Version 3

Martin L. Jones, Electron Microscopy Science Technology Platform, The Francis Crick Institute, London, UK

4 Views Cite this report Responses(0)

Approved

The addition of more novice-focused documentation is a good addition.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Image Analysis, CLEM, Machine Learning, Electron microscopy

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

20 Views

29 Jun 2023 | for Version 2

Christopher Schmied, AG Han Sun, Leibniz-Forschungsinstitut für Molekulare Pharmakologie, Berlin, Berlin, Germany

20 Views Cite this report Responses(1)

Approved With Reservations

A more extensive methods section. Describing clearer what the architecture was and any important or changed hyperparameters. Also, the hardware and the training time should be specified.
Improve documentation from the perspective of a novice user.
State how users should get in touch for support.

Is the rationale for developing the new software tool clearly explained?

Yes
Is the description of the software tool technically sound?

Yes
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Data science, bioimage analysis, cell biology.

Respond to this report

Responses (1)

Author Response

13 Apr 2024

Philip Kollmannsberger, Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany

Authors: Thank you for the detailed evaluation and feedback – we greatly appreciate your time and effort!

Reviewer: I found some points problematic in the documentation:
"The EM image should look like the testing EM.png." This description is too unspecific for novice users. Important would be here to emphasize that this is in context to the application of the pre-trained model on new data.

Authors: We now improved the documentation and tried to make it more accessible for novice users. The instructions on the requirements for the images to be aligned are now more detailed, and a better explanation on when to train a custom model was included.

Reviewer: Great that the conda .yml is provided for easy installation of the dependencies. As a warning, I experienced that the creation of the conda environment took a long time on my machine. But could be specific to my setup. The possibilities of my review here are also limited since I do not have the possibility to generate my own training data from scratch and I can only test the provided data. Within these constraints, I was able to test the notebooks for their correct operation.

Authors: To improve the installation of the dependencies for network training, we optimized the .yml file for the conda environment. Furthermore, we included mamba as package manager in the documentation to speed up solving the environment. The modified environment file was tested under Windows, Mac M1 and Linux.

Reviewer: The methods section of the article is unfortunately a bit sparse. The actual network architecture is only mentioned in the results. No further detail is provided. It is fine if the default values of the out-of-the-box network are used. However, this should be clearly stated in the methods. I would also wish a better description of how the training has been performed. Including how long the training took and on which hardware this was performed.

Authors: We updated the methods section and added a new paragraph to include information about the training parameters, the hardware that was used and how long the training took. The new paragraph at the end of the “Implementation” subsection reads:

“We trained the ProjectionCARE 2D UNet architecture from CSBDeep12 with 3 layers and a batch size of 8 for 150 epochs, using the default parameters of CSBDeep unless stated otherwise. Training on a GPU took between one (RTX 2080 Ti, cuda 11.3, Windows 10) and three hours (Tesla P100, cuda 11.7, Ubuntu 20.04) and around 10 hours on a CPU (Intel i7-8700K at 3.70 GHz, Windows 10). The Jupyter notebook for training on custom datasets as well as documentation how to use it are available online (see software availability statement).”

Reviewer: I really enjoyed that the authors provide all necessary code and example images on their GitHub repo. The only tiny thing missing is to clearly state their preferred way of users getting in touch for support. I would suggest clearly stating this in the software availability section.

Authors: This information was indeed missing, we now include contact information for support both in the manuscript as well as in the documentation. The software availability section now contains information on how to contact the authors for support and questions. We also added a more detailed section on how to get in touch to the beginning of the documentation at the repository.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

11 Views

30 Aug 2022 | for Version 2

Martin L. Jones, Electron Microscopy Science Technology Platform, The Francis Crick Institute, London, UK

11 Views Cite this report Responses(0)

Approved

No further comments, the authors have covered all of the comments very satisfactorily.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Image Analysis, CLEM, Machine Learning, Electron microscopy

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

44 Views

08 Feb 2021 | for Version 1

Reinhard Rachel, Centre for Electron Microscopy, Faculty of Biology and Preclinical Medicine, University of Regensburg, Regensburg, Germany

44 Views Cite this report Responses(1)

Approved With Reservations

The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere?
What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed? - the authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow?
Histogram equalization of EM images - was this done with routines implemented in Fiji?
Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM? or at least comment on this?

With comments on my points listed above, and according improvements in the manuscript, this can be indexed - thus, approved with reservations.

Is the rationale for developing the new software tool clearly explained?

No
Is the description of the software tool technically sound?

Partly
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

TEM, STEM, tomography, CLEM; microbiology, cell biology

Respond to this report

Responses (1)

Author Response

16 Aug 2022

Philip Kollmannsberger, Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany

R: The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere?

A: If the structures used for alignment (e.g., chromatin) can be resolved and the WF images can be manually aligned, it is also possible to obtain a prediction and do automated alignment, although the achievable quality of the registration is obviously limited by the resolution of the fluorescence channel. We now added this in the discussion of the scope and limitations of the tool.

R: What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed?

A: Using a trained network for prediction and alignment can be done fully automatically in Fiji, only the paths of the images and network must be provided in the user dialog. For retraining, some knowledge (e.g., setting up a python environment) is necessary. We now added a more detailed description of the training workflow. Currently, the training process should probably be performed by some more experienced person while using the trained network does not require any special knowledge.

R: The authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow?

A: It may be possible to use the provided network trained on our data on images from other labs directly, however in most cases re-training is required. We now describe the required workflow in more detail and perform a quantitative comparison of registration quality where images from an experiment were either included in the training set or not.

R: Histogram equalization of EM images - was this done with routines implemented in Fiji?

A: We used the standard histogram equalization implemented in Fiji. We added this detail to the method description.

R: Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM? or at least comment on this?

A: We now comment on ec-CLEM and its “autofinder” functionality which is closest to our tool in terms of functionality. While ec-CLEM autofinder uses spot finding or centers of segmented regions as points of interest to perform registration on, we predict the image of the other modality and then use correlation-based alignment.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

57 Views

04 Feb 2021 | for Version 1

Martin L. Jones, Electron Microscopy Science Technology Platform, The Francis Crick Institute, London, UK

57 Views Cite this report Responses(1)

Approved With Reservations

In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.
The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo?
Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.
I believe the preferred capitalisation is "Fiji" and not "FIJI"
ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.
For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.
In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.
If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists.
The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.
An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial!
The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well.
Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract.
The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful

Is the rationale for developing the new software tool clearly explained?

Yes
Is the description of the software tool technically sound?

Partly
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Image Analysis, CLEM, Machine Learning, Electron microscopy

Respond to this report

Responses (1)

Author Response

16 Aug 2022

Philip Kollmannsberger, Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany

R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.

A: We removed the mention of a specific stain in the abstract to reflect the general applicability of the method.

R: The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo?

A: The code is a Jython script with graphical user interface to be run within Fiji. From the user's perspective, it is indeed a plugin. We changed “macro” to “plugin”, as suggested.

R: Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.

A: The trained network can be applied to different datasets if they have similar contrast, usually from the same microscope or sample type. Otherwise, (re-)training of the network is necessary. We now make this clearer in the manuscript and include a description of the training workflow.

R: I believe the preferred capitalisation is "Fiji" and not "FIJI"

A: Thank you, we corrected this.

R: ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.

A: We now refer to ec-CLEM autofinder which is similar in terms of functionality. It is based on finding corresponding features in both modalities, e.g., using wavelet-based spot detection or the centers of segmented regions, and then performing registration of these automatically detected landmarks. We did not perform a direct quantitative comparison. If suitable corresponding spots can be found (e.g. beads), the performance on the same images using the same registration algorithm should be comparable. In cases where spot finding or similar methods in ec-CLEM do not perform well because the images are too dissimilar or if there are no fiducial markers, then transferring the EM image to fluorescence using DeepCLEM could be used to generate input data for point-based registration in ec-CLEM.

R: For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.

A: We agree that these methods are more reproducible and added an additional note to the paragraph on manual registration to encourage users to use such reproducible methods. Since we already had created the training data, we did, however, not repeat the alignment. For assessing the performance (see below), we instead artificially displaced the two channels - in this case, the steps and transforms are also known exactly.

R: In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.

A: We now include more detail about the training workflow in the implementation section and compared the performance for different amounts of training images.

R: If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists.

A: We extended both the description of the training parameters as well as the documentation of the entire workflow, including training in the Jupyter notebook.

R: The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.

A: The text now explicitly mentions the use of the "Register Virtual Stack Slices" plugin and the "similarity" transform in our work, but in principle, our workflow should also work with other means of correlation-based registration.

R: An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial!

A: We now performed quantification of the quality of the registration using the manually aligned ground truth data as a reference where the best solution is already known. We observed that in most cases, the registration either works and has a very small error or is completely off by several 100 nm, which should be easy to spot. If a few images from an experiment are manually aligned and included in the training set, the registration of the remaining images improves significantly (Table 1).

Instead of providing a confidence metric, which would indeed be non-trivial, we recommend visual inspection using the overlay. Possible ways to implement a confidence metric would be to output a confidence level for the neural network prediction, and to check the quality of the alignment using a correlation metric between fluorescent and predicted image.

R: The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well.

A: We added more detail regarding the scope and limitations of this tool and its current requirements to the abstract, introduction, and discussion section, including the need for at least three heterochromatin patches if chromatin staining is used for registration.

R: Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract. The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful

A: The way it is currently configured, DeepCLEM can in principle be used to register any CLEM data where corresponding 2D slices are available in both modalities, whether they were acquired on the same cut sections (srAT) or prior and post sectioning. The examples we provide however are restricted to srAT. We now state this in the text.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Markert SM, Britz S, Proppert S, et al.: Filling the gap: adding super-resolution to array tomography for correlated ultrastructural and molecular identification of electrical synapses at the C. elegans connectome. Neurophotonics. 2016; 3(4): 041802. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Markert SM, Bauer V, Muenz TS, et al.: 3D subcellular localization with superresolution array tomography on ultrathin sections of various species. Methods Cell Biol. 2017; 140: 21–47. PubMed Abstract | Publisher Full Text

[3] 3. García-Fernández E, Koch G, Wagner RM, et al.: Membrane Microdomain Disassembly Inhibits MRSA Antibiotic Resistance. Cell. 2017; 171(6): 1354–1367.e20. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Cardona A, Saalfeld S, Schindelin J, et al.: TrakEM2 Software for Neural Circuit Reconstruction. PLoS One. 2012; 7(6): e38011. PubMed Abstract | Publisher Full Text | Free Full Text

[5] 5. Paul-Gilloteaux P, Heiligenstein X, Belle M, et al.: eC-CLEM: flexible multidimensional registration software for correlative microscopies. Nat Methods. 2017; 14(2): 102–103. PubMed Abstract | Publisher Full Text

[6] 6. von Chamier L, Laine RF, Henriques R: Artificial intelligence for microscopy: what you should know. Biochem Soc Trans. 2019; 47(4): 1029–1040. PubMed Abstract | Publisher Full Text

[7] 7. Belthangady C, Royer LA: Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat Methods. 2019; 16(12): 1215–1225. PubMed Abstract | Publisher Full Text

[8] 8. Christiansen EM, Yang SJ, Ando DM, et al.: In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images. Cell. 2018; 173(3): 792–803.e19. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Guo SM, Yeh LH, Folkesson J, et al.: Revealing architectural order with quantitative label-free imaging and deep learning. eLife. 2020; 9: e55502. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Schindelin J, Arganda-Carreras I, Frise E, et al.: Fiji: an open-source platform for biological-image analysis. Nat Methods. 2012; 9(7): 676–682. PubMed Abstract | Publisher Full Text | Free Full Text

[11] 11. Uçeyler N, Kafke W, Riediger N, et al.: Elevated proinflammatory cytokine expression in affected skin in small fiber neuropathy. Neurology. 2010; 74(22): 1806–1813. PubMed Abstract | Publisher Full Text

[12] 12. Weigert M, Schmidt U, Boothe T, et al.: Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat Methods. 2018; 15(12): 1090–1097. PubMed Abstract | Publisher Full Text

[13] 13. Virtanen P, Gommers R, Oliphant TE, et al.: SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods. 2020; 17(3): 261–272. PubMed Abstract | Publisher Full Text | Free Full Text

[14] 14. Abadi M, Agarwal A, Barham P, et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. arXiv: 160304467 [cs]. 2016. Publisher Full Text

[15] 15. Pielawski N, Wetzer E, Öfverstedt J, et al.: CoMIR: Contrastive Multimodal Image Representation for Registration. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., 2020; 33: 18433–18444. Publisher Full Text

[16] 16. Seifert R: CIA-CCTB/Deep_CLEM: First release of DeepCLEM (Version v1.0). Zenodo. 2020. http://www.doi.org/10.5281/zenodo.4095247

DeepCLEM: automated registration for correlative light and electron microscopy using deep learning

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Methods

Data acquisition

Manual registration

Implementation

Figure 1. Schematic of the "DeepCLEM" workflow.

Operation

Figure 2. GUI and input parameters for "DeepCLEM".

Results

Comparison of network architectures

Optimization of preprocessing

Quantitative evaluation

Table 1. Quantitative evaluation.

Discussion

Data availability

Software availability

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated