A multi-spectral myelin annotation tool for machine learning based myelin quantification

Myelin is an essential component of the nervous system and myelin damage causes demyelination diseases. Myelin is a sheet of oligodendrocyte membrane wrapped around the neuronal axon. In the fluorescent images, experts manually identify myelin by co-localization of oligodendrocyte and axonal membranes that fit certain shape and size criteria. Because myelin wriggles along x-y-z axes, machine learning is ideal for its segmentation. However, machine-learning methods, especially convolutional neural networks (CNNs), require a high number of annotated images, which necessitate expert labor. To facilitate myelin annotation, we developed a workflow and software for myelin ground truth extraction from multi-spectral fluorescent images. Additionally, to the best of our knowledge, for the first time, a set of annotated myelin ground truths for machine learning applications were shared with the community.


Open Peer Review
Any reports and responses or comments on the article can be found at the end of the article.

Introduction
Myelin degeneration causes neurodegenerative disorders, such as multiple sclerosis (MS) 1,2 .There are no remyelinating drugs.Myelin quantification is essential for drug discovery, which often involves screening thousands of compounds 3 .Currently, fluorescent myelin quantification is manual, and labor-intensive.Automation of quantification using machine learning can facilitate drug discovery by reducing time and labor costs 4 .However, myelin annotation suffers the same limitations as manual quantification.To assist researchers and bioimage analysts, we developed a workflow and software for myelin ground truth extraction from multi-spectral fluorescent images.
Myelin is formed by oligodendrocytes wrapping the axons 5 .It is identified by continuous co-localization of cellular extensions that span multiple channels and z-sections (Figure 1).Note that, the continuity is in the eye of the expert while myelin appears granular in digitized images due to the nature of staining.This necessitates the annotation to be pixel-based and th expert to fill the gaps making the process very laborious.In our workflow, co-localizing pixels, candidate myelins, were determined using Computer-assisted Evaluation of Myelin (CEM) software that we previously developed 6 .In this context, CEM software functions as a candidate myelin detection program because it simply identifies overlapping pixels.Briefly, CEM removes cell bodies, defined as the overlap of nuclei and cellular marker, and identifies overlapping pixels between remaining oligodendrocyte and neuron channels 6 .
In the current study, the CEMotate tool 7 was developed to efficiently evaluate these candidate myelins and to extract myelin ground truths.Using CEMotate, an RGB-composite z-section image, corresponding CEM output image, and expert's markings can be visualized simultaneously to decide whether to keep or remove candidate pixels (see Implementation).
The user can move along x-y-z axes and show/hide channels, images, and markings.Markings from the -1/+1 z-sections can be viewed simultaneously.Finally, CEMotate enables two experts to independently mark myelin at different times and on different computers.When the files containing their myelin markings are shared and overlaid, it allows for the simultaneous visualization of both experts' annotations.This feature is crucial for inter-expert comparisons.
Using the described workflow, we annotated five images encompassing approximately 2 × 8 mm by 30-50 μm volume.
The entire process, which would have taken several weeks, took approximately 5 days.More than 30,000 feature images were extracted from these five images and were used for testing various machine-learning methods [8][9][10] .The annotated images, which are available with the manuscript, are a resource for the researchers working not only on myelin detection but also on segmenting multi-spectral images.

Image acquisition
Images were previously acquired 6 .Briefly, co-cultures of mouse embryonic stem cell-derived oligodendrocytes and neurons were grown in microfluidic chambers.After myelin formation, cells were fixed in paraformaldehyde and were stained with 1:1,000 mouse or rabbit anti-TUJ1 (Covance), 1:50 rat anti-MBP (Serotec), and DAPI (Sigma).Images were acquired on Zeiss confocal microscopes as tiles approximately 2mm×8mm.The z-axis, 30-50 μm, was covered by 1-μm-thick optical z-sections.The tiles were stitched together on Zen software (Zeiss).No further processing was done.

Implementation
In CEMotate, a new project is started by loading oligodendrocyte, axon, and nucleus images, red, green, and blue channels respectively in the example (Figure 2).Users can save and reopen projects.In CEMotate, users can zoom using the mouse wheel and can move in the x-y axes and z-axis using scroll bars and buttons respectively (Figure 2 and Figure 3).
Myelin pixels may be marked at various thickness values (Figure 3).CEMotate records myelin drawings as vectors in the ".iev" files.These vectors can be modified or deleted in CEMotate (Figure 3).Optionally, to facilitate myelin detection, the candidate myelins can be loaded from CEM 6 or another source that generates binary images of myelin markings.Myelin identification using CEM is described in detail in 6.Output of CEM, is a binary image, which is converted to vectors using the included module (Figure 4).Note that the conversion overwrites existing myelin vectors.
Additionally, myelin regions from two sources can be visualized simultaneously.This allows visualization of myelins annotated by experts and CEM, to do so, first, rename and copy the ''.iev'' file containing second myelin vectors to the same folder.Next, modify the ''.ini" files as shown in Figure 5.After loading the modified ''.ini" file using the 'Merge Edit' button, myelin vectors will be shown in two different colors (Figure 6).These vectors can be modified as in Figure 6.
Once done with marking, users can convert the myelin vectors into an image using the "Save Myelin Mask Image" button.We implemented this strategy to extract gold standard myelin ground truths.

Comparative analysis
The myelins marked by two experts were compared against the gold standards.Experts' precision for each image was calculated as described in 9.The average precision was calculated as mean of precision values of each expert for each image.

Operation
CEMotate is written in Pascal with the Delphi XE5 platform.
The program can be run on 64-bit Microsoft Windows operating systems.

Results
In this study, myelin were marked by one moderately qualified and one entry level experts, on previously acquired oligodendrocyte and neuron co-culture images 6 using the described workflow (see Implementation).A third, highly qualified expert evaluated their markings and extracted gold standard myelin ground truths.The ground truth images were saved as TIF on CEMotate 7 .All images are available (see below).While CEM determined the candidate myelins on five images in approximately 43 minutes, ML approach took only 1.04 seconds for the same process 8 (Table 1).Extracting the gold standard myelin ground truths from five images with candidate pixels that were determined by CEM took approximately another 35 hours for one expert.This process involved determining FPs and FNs on ImageJ.The same process took approximately 20 hours for an expert using CEMotate.Thus, over 40% of time was  saved (Table 2).Moreover, CEMotate enabled collaboration of three experts for accelerated myelin ground truth extraction.Because ImageJ does not have such a feature, we could not directly compare the times saved for this process.
CEM identified 219032 candidate myelin pixels on five images.Two experts identified TP myelins.A third expert evaluated these results to obtain the gold standard myelin ground truths which covered 9550 pixels.To the best of our knowledge, this is the first time myelin ground truths of fluorescent images are shared with the science community.
Next, we calculated each expert's performance (Table 3).Two experts averaged 48.39% precision.The highest precision of an expert was 87.95% for one image.In comparison, our customized-CNN and Boosted Trees approaches, which were trained using ground truths images using the data annotated with CEM consistently reached precision values over 99% 8 .These results suggest that, machine learning methods can outperform human annotators once trained with accurately labeled data.

Conclusion
CEMotate 7 accelerates annotation of multi-spectral images.As an example, we used it to annotate myelin, which can only be identified as co-localization of neuron and oligodendrocyte membranes within certain criteria.CEMotate's visualization features simplified inter-expert collaboration and validation.Moreover, myelin ground truths accompanying this manuscript are a resource for the researchers working on segmenting myelin and other features in multi-spectral images.

Time (~) 35 hours 20 hours
Overall, the article is clearly written and the tool is well motivated.The ability for experts to work collaboratively on the same data is an important aspect of the new tool that might both accelerate annotation and improve the quality of the annotations.
I think the manuscript could be improved in the following directions to be more informative: The results section mentions three experts without qualifying their expertise level.I think it would be better to include their area and level of expertise to allow the reader better interpret the tabulated results.

○
The training data used to the custom-CNN/Boosted Tree approach is not immediately clear.Is it trained using the data annotated with the proposed tool or with CEM?The reference [8]  indicates the latter but I think it would be good to mention this explicitly.

○
The collaborative editing aspect could be expanded if it allows interaction between experts beyond simple parallelism.The level of collaboration allowed by the tool is not clear from the text.

○
Few minor comments: Second paragraph in the Results section (singe sentence) is redundant since the Methods section already provides the same information.

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?Yes Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Computational neuroscience, structural studies of white matter, dynamical models of glial membrane.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Version 2
Reviewer Report 08 April 2022 https://doi.org/10.5256/f1000research.121217.r126756 © 2022 Janjic P.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Predrag Janjic
Research Center for Computer Science and Information Technology, Macedonian Academy of Sciences and Arts, Skopje, North Macedonia The authors have addressed the issues within the initial review carefully.
I would suggest that the introduction stresses clearly that the specific nature of myelin identification and annotation difficulties this work addresses matter specifically for fluorescent imaging due to the particular nature of the stain and the imaging itself.The legacy EM studies do not meet the same problems necessarily, principally the granular appearance of myelin due to the nature of localization of the fluorophore.The granularity of myelin in this imaging is the critical feature making the annotation of these images pixel/particle based and very laborious.
The concluding statement in the following paragraph (bold), appearing in the Abstract as well: "CEM identified 219032 candidate myelin pixels on five images.Two experts identified TP myelins.A third expert evaluated these results to obtain the gold standard myelin ground truths which covered 9550 pixels.To the best of our knowledge, this is the first time myelin ground truths are shared with the science community." I believe this is correct just for fluorescent imaging, since annotated EM images have been shared, at least within data archives and along qualified requests after the availability has been stated in the papers.
Will appreciate if authors would fix this at both places.

Is the rationale for developing the new software tool clearly explained? Yes
Is the description of the software tool technically sound?Yes

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?Yes Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Computational neuroscience, structural studies of white matter, dynamical models of glial membrane.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
The manuscript introduces a 3D extension of myelin annotation tool reported in Ref [5], now applicable to fluorescent stacks.Although the main rationale to develop an automated tool for producing ground truth images is clear, and the importance has been laid out, methodological details and comparative data are missing for a researcher dealing with myelin imaging to get an idea of what the tool is really computationally doing in order to decide on its practical utility.
Please rework the Introduction and despite rather harsh length constraints add some minimal description of what algorithms in Ref [5] do.

Image acquisition:
Extend on the image and image processing details like the size of the captures, pixel size, deconvolving or not, depth corrections (you have some of it in the Results).

Implementation:
Please extend on what has been done to integrate CEM tool, and parallel visualization of myelin in subsequent planes, figures Fig. 5 and Fig. 6., elaborating the utility of this step which is the main procedural added value of the presented tool.Please move to additional material or remove Fig. 1 -Fig.4., which are more of a user guide and are disruptive in the value presentation.

○
Comparative analysis: Some more data is needed, new or from Ref [8] for a reader to be able to get the overall impression.Please consider adding a Benchmarking paragraph where you would extend a bit on some benchmarking given within the Results, with an estimate (table) of the computational time needed for the CEM scope per slice, and the total time to process a whole stack, all in order a potential end user to get an impression of the effort needed.

Results:
Please extend on limitations & issues of the CEM3D, and try to estimate if possible specific performance over a whole stack using this extension.(which is the actual improvement and a gain compared to relying only on CEM).

Is the rationale for developing the new software tool clearly explained? Yes
Is the description of the software tool technically sound?No

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Partly
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?Partly Competing Interests: No competing interests were disclosed.
Reviewer Expertise: computational neuroscience, structural studies of white matter, dynamical models of glial membrane.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.
Author Response 23 Feb 2022

Bilal Kerman
We thank Predrag Janjic for his helpful comments.We believe that we address his concerns and the updated manuscript is easier to read and more satisfactory to the readers.Please see our point by point responses below.
I apologize in the time it took us to update the manuscript.I moved between institutions and it took me a long time to settle in part due to the pandemic restrictions.
The manuscript introduces a 3D extension of myelin annotation tool reported in Ref [5], now applicable to fluorescent stacks.Although the main rationale to develop an automated tool for producing ground truth images is clear, and the importance has been laid out, methodological details and comparative data are missing for a researcher dealing with myelin imaging to get an idea of what the tool is really computationally doing in order to decide on its practical utility.

Please rework the Introduction and despite rather harsh length constraints add some minimal description of what algorithms in Ref[5] do.
Response 1: We updated the Introduction to include: "In this context, CEM software functions as a candidate myelin detection program because it simply identifies overlapping pixels.Briefly, CEM removes cell bodies, defined as the overlap of nuclei and cellular marker, and identifies overlapping pixels between remaining oligodendrocyte and neuron channels 6 ."

Image acquisition:
Extend on the image and image processing details like the size of the captures, pixel size, deconvolving or not, depth corrections (you have some of it in the Results).
Response 2: The tool described in this publication is not an image processing tool per se.It is main function is annotation of myelin pixels for machine learning studies.Therefore, in this revision, to prevent any confusion, we renamed the tool as CEMotate.Response 3: In this study, the CEM tool is not integrated directly to CEMotate.We used images previously processed by CEM and annotated them using CEMotate.The details of processing by CEM were given in the reference Kerman et al. 2015.The one major goal of this manuscript is to introduce our myelin annotation tool and to describe how to utilize it.Therefore, we that Fig.

Results:
Please extend on limitations & issues of the CEM3D, and try to estimate if possible specific performance over a whole stack using this extension.(which is the actual improvement and a gain compared to relying only on CEM).
We updated the Results section to include the metrics as described above.
Competing Interests: No competing interests were disclosed.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com

Figure 1 .
Figure 1.An example of multi-spectral fluorescent image.20× confocal microscopy image tiles were stitched together covering approximately 2 × 8 mm by 30-50 μm volume.Boxed area is enlarged to show myelin (brackets) and the false positive pixels (circles).

Figure 2 .
Figure 2. Starting a new project in CEMotate.Buttons for loading oligodendrocyte, axon, and nucleus images, and navigating the z-stack button to up and down are marked.

Figure 3 .
Figure 3. Myelin drawing and saving in CEMotate.The relevant buttons and myelin vectors are marked.

Figure 4 .
Figure 4. Loading CEM output image.To load candidate myelin pixels, use "Convert Binary Image to Vector" button.

Figure 6 .
Figure 6.Modifying the myelin vectors.CEM candidate myelins or two experts' markings can be shortened, deleted or drawn over.

Figure 5 .
Figure 5. Visualizing two myelin vectors simultaneously.Modify .inifile as in the lower panels and load it using "Merge Edit" button.

○
Some sentences in the Implementation section are phrased in the tone of a user's manual.They should be rephrased to be consistent with the style of the article."conversion will overwrite *your* existing myelin vectors" -> "conversion overwrites existing myelin vectors" ○ Is the rationale for developing the new software tool clearly explained?YesIs the description of the software tool technically sound?YesAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?PartlyIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?PartlyAre the conclusions about the tool and its performance adequately supported by the findings presented in the article?Yes Competing Interests: No competing interests were disclosed.Reviewer Expertise: Computer vision, image classification, object detection and tracking Is the rationale for developing the new software tool clearly explained?Yes Is the description of the software tool technically sound?Yes Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?Yes

:
Please extend on what has been done to integrate CEM tool, and parallel visualization of myelin in subsequent planes, figures Fig.5 and Fig.6., elaborating the utility of this step which is the main procedural added value of the presented tool.Please move to additional material or remove Fig.1 -Fig.4., which are more of a user guide and are disruptive in the value presentation.

Some more data is needed, new or from Ref[8] for a reader to be able to get the overall impression. Please consider adding a Benchmarking paragraph where you would extend a bit on some benchmarking given within the Results, with an estimate (table) of the computational time needed for the CEM scope per slice, and the total time to process a whole stack, all in order a potential end user to get an impression of the effort needed. Response 4: The updated text below:
1 -Fig.4.are useful for readers.ML approach took only 1.04 seconds for the same process 8 (Table1).Extracting the gold standard myelin ground truths from five images with candidate pixels that were determined by CEM took approximately another 35 hours for one expert.This process involved determining FPs and FNs on ImageJ.The same process took approximately 20 hours for an expert using CEMotate.Thus, over 40% of time was saved (Table2).Moreover, CEMotate enabled collaboration of three experts for accelerated myelin ground truth extraction.Because ImageJ does not have such a feature, we could not directly compare the times saved for this process.

Table 2 . Time comparison for ImageJ and CEMotate annotation
candidate myelin pixels on five images.Two experts identified TP myelins.A third expert evaluated these results to obtain the gold standard myelin ground truths which covered 9550 pixels.To the best of our knowledge, this is the first time myelin ground truths are shared with the science community.

Table 3 . Experts' average precisions on candidate myelin pixels of five images Expert 1 Expert 2
8alculated each expert's performance.Two experts averaged 48.39% precision.The highest precision of an expert was 87.95% for one image.In comparison, our customized-CNN and Boosted Trees consistently reached precision values over 99%8.These results suggest that machine learning methods can outperform human annotators once trained with accurately labeled data.