DeepCLEM: automated registration for correlative light and electron microscopy using deep learning

R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step.

A: We removed the mention of a specific stain in the abstract to reflect the general applicability of the method.

R: The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo?

A: The code is a Jython script with graphical user interface to be run within Fiji. From the user's perspective, it is indeed a plugin. We changed “macro” to “plugin”, as suggested.

R: Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset.

A: The trained network can be applied to different datasets if they have similar contrast, usually from the same microscope or sample type. Otherwise, (re-)training of the network is necessary. We now make this clearer in the manuscript and include a description of the training workflow.

R: I believe the preferred capitalisation is "Fiji" and not "FIJI"

A: Thank you, we corrected this.

R: ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible.

A: We now refer to ec-CLEM autofinder which is similar in terms of functionality. It is based on finding corresponding features in both modalities, e.g., using wavelet-based spot detection or the centers of segmented regions, and then performing registration of these automatically detected landmarks. We did not perform a direct quantitative comparison. If suitable corresponding spots can be found (e.g. beads), the performance on the same images using the same registration algorithm should be comparable. In cases where spot finding or similar methods in ec-CLEM do not perform well because the images are too dissimilar or if there are no fiducial markers, then transferring the EM image to fluorescence using DeepCLEM could be used to generate input data for point-based registration in ec-CLEM.

R: For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training.

A: We agree that these methods are more reproducible and added an additional note to the paragraph on manual registration to encourage users to use such reproducible methods. Since we already had created the training data, we did, however, not repeat the alignment. For assessing the performance (see below), we instead artificially displaced the two channels - in this case, the steps and transforms are also known exactly.

R: In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model.

A: We now include more detail about the training workflow in the implementation section and compared the performance for different amounts of training images.

R: If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists.

A: We extended both the description of the training parameters as well as the documentation of the entire workflow, including training in the Jupyter notebook.

R: The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives.

A: The text now explicitly mentions the use of the "Register Virtual Stack Slices" plugin and the "similarity" transform in our work, but in principle, our workflow should also work with other means of correlation-based registration.

R: An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial!

A: We now performed quantification of the quality of the registration using the manually aligned ground truth data as a reference where the best solution is already known. We observed that in most cases, the registration either works and has a very small error or is completely off by several 100 nm, which should be easy to spot. If a few images from an experiment are manually aligned and included in the training set, the registration of the remaining images improves significantly (Table 1).

Instead of providing a confidence metric, which would indeed be non-trivial, we recommend visual inspection using the overlay. Possible ways to implement a confidence metric would be to output a confidence level for the neural network prediction, and to check the quality of the alignment using a correlation metric between fluorescent and predicted image.

R: The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well.

A: We added more detail regarding the scope and limitations of this tool and its current requirements to the abstract, introduction, and discussion section, including the need for at least three heterochromatin patches if chromatin staining is used for registration.

R: Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract. The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful

A: The way it is currently configured, DeepCLEM can in principle be used to register any CLEM data where corresponding 2D slices are available in both modalities, whether they were acquired on the same cut sections (srAT) or prior and post sectioning. The examples we provide however are restricted to srAT. We now state this in the text.