ROOSTER: An image labeler and classifier through interactive recurrent annotation

Zhou Tang; Yang Hu; Zhiwu Zhang

doi:10.12688/f1000research.127953.1

Home Browse ROOSTER: An image labeler and classifier through interactive recurrent...

ALL Metrics

-

Views

-

Downloads

Get PDF

Get XML

Export

▬

✚

Software Tool Article

ROOSTER: An image labeler and classifier through interactive recurrent annotation

[version 1; peer review: 1 approved with reservations]

Zhou Tang¹, Yang Hu ¹, Zhiwu Zhang¹

PUBLISHED 06 Feb 2023

Author details Author details

¹ Department of Crop and Soil Sciences, Washington State University, Pullman, Washington, 99163, USA

Zhou Tang
Roles: Formal Analysis, Investigation, Validation, Writing – Review & Editing

Yang Hu
Roles: Methodology, Software, Visualization, Writing – Review & Editing

Zhiwu Zhang
Roles: Conceptualization, Funding Acquisition, Supervision, Writing – Original Draft Preparation

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Plant Science gateway.

This article is included in the Artificial Intelligence and Machine Learning gateway.

This article is included in the Python collection.

Abstract

A large amount of training data is usually lacking at the beginning of system development and labeling such a large number of RGB (red, green, blue) images is laborious. Interactive recurrent annotation is beneficial to incrementally gain training images in the stream of the system development and provides an opportunity to reduce human workload. We developed a software package, ROOSTER, to integrate both labeling and prediction in a single user-friendly graphic user interface with interactive deep learning to reduce the laborious human labeling for fast development of machine vision systems. Predictions can be performed under both single-image mode and batch mode for multiple images. The prediction results can be used as the initial image labeling and manually adjusted under a single image mode. Human labeling and machine predictions are visualized on the same image. ROOSTER provides fully automatic labeling for abundantly available initial images of wheat stripe rust to gain essential predictability. The navigation of integrating prediction with labeling benefits human adjustment to iteratively improve predictability. The development of a detection system for wheat stripe rust was presented as a use case to demonstrate the efficiency of using interactive deep learning to develop machine vision systems.

Keywords

Machine vision, deep learning, labeling, classification, software

Corresponding authors: Yang Hu, Zhiwu Zhang

Competing interests: No competing interests were disclosed.

Grant information: This project was partially supported by United States Department of Agriculture (USDA) National Institute of Food and Agriculture (NIFA) (Awards 2018-70005-28792, 2019-67013-29171 and 2020-67021-32460), and the Washington Grain Commission (Endowment, 126593 and 134574), which were assigned to Zhiwu Zhang.

Copyright: © 2023 Tang Z et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Tang Z, Hu Y and Zhang Z. ROOSTER: An image labeler and classifier through interactive recurrent annotation [version 1; peer review: 1 approved with reservations]. F1000Research 2023, 12:137 (https://doi.org/10.12688/f1000research.127953.1) First published: 06 Feb 2023, 12:137 (https://doi.org/10.12688/f1000research.127953.1) Latest published: 06 Feb 2023, 12:137 (https://doi.org/10.12688/f1000research.127953.1)

Introduction

Image recognition is a critical part of machine vision. It identifies objects in images, such as specific types of tumor cells or leaves infected by a specific pathogen. Due to the complexity of solving such problems, deep learning (DL) is leveraged to achieve the required results. The challenge of DL is the requirement of a large number of training images before the learning task.¹ Because training data is not always available at the beginning, and the new data arrives in the format of sequential stream.² Furthermore, labeling a large number of images is labor-intensive.³ To reduce the time consumption of laborious human labeling, it is ideal to incrementally label images to train a DL system and let the system label new images.⁴^,⁵ After human corrections, the new labels can enhance the system further until the required results are achieved.⁶^,⁷ We developed an open-source Python package named ROOSTER, to fit such a need. ROOSTER not only has a friendly graphic user interface to label images or their sub-images but also accepts any pre-trained models in the format of PyTorch for classification. We demonstrated the usage of ROOSTER in developing a machine vision system to detect wheat stripe rust using images from smartphones and drones.

Methods

Implementation

The interface of ROOSTER contains a control panel on the bottom and a display panel on the top. An image is imported by clicking the Image button. The number of rows and columns can define tiles. The grids can be hidden or displayed by clicking the Grid button. The image and grids can be zoomed in (+), zoomed out (-), or moved by clicking and dragging. When numbers of rows and columns are defined, the statuses of tiles are set to control by default, indicated by white lines on the top left corners. With a double-click on a tile, its status can be switched between the default status and the alternative status indicated by red lines in the top left corner (see Figure 1). The statuses of all tiles can be reversed by clicking the Reverse button.

Figure 1. ROOSTER graphic user interface and its automatic and semi-automatic image labeling through classification.

The case image was for wheat stripe rust which can be found in Underlying data.¹¹

ROOSTER can load ResNet-18 based neural networks model to make predictions. Prediction is applied to all tiles by loading a pre-trained model or a model trained with part of user-owned images. For example, RustNet is a ResNet-18 based neural network that can be used for wheat stripe rust detection.⁸ It was pre-trained with wheat stripe rust images from different situations, which can be loaded into ROOSTER to make predictions. Visualization of prediction on a tile is based on its current status. The disagreement between the prediction and the current status is indicated by a dot on the top left of the tile. The dot is in red if the current status is in the default status and white otherwise (Figure 1). ROOSTER outputs include a PNG file for an overview of labeling, images of cropped tiles, and an Excel file (Map) to indicate the statuses of individual tiles. The output Map file can also be used as the input to define the status of an image. This function allows users to save label results and resume labeling later.

ROOSTER can process images in two modes: batch mode and single image mode. When the batch mode is checked, clicking Image button defines the image folder. Otherwise, the Image button clicking chooses a single image. Under single image mode, an option is available to use the Map button to apply a pre-classification result (an Excel form) to the current image, including the numbers of rows and columns and statuses of all tiles. If no excel file is attached, users need to define the total number of tiles by specifying the number of rows and columns and click the Grid button to draw the grids. Users can switch the status between the default and the alternative for each tile. Prediction can be performed for either multiple images with the batch mode or the single image by clicking the Predict button. In either case, a window will pop out for users to attach a classification model, e.g., RustNet.pth. The labels can be saved by clicking the Export button. The exported result can be imported through the Map button to continue the customization of labels.

Operation

ROOSTER is developed with Python 3.6 for 64-bit processors on Mac OS, Linux, and Windows with a minimal 16 GB memory (the code is available in Software availability¹²). When numbers of rows and columns are defined, ROOSTER can load ResNet-18-based models to predict tiles before human labeling is involved. Human laborers can correct labels by double-clicking the tile. The disagreement between the prediction model and human labeling would be shown with a two-color dot in the top left corner. With ROOSTER, users can start with labeling part of images and train the initial version of the prediction model. The initial version of the model can be reloaded to predict the rest of images in the ROOSTER and retrain the model with the updated dataset, which could increase the labeling efficiency and gradually improve the model accuracy.

Use case

We used ROOSTER to automatically label 200 images of plants containing no infection as the default status and 200 images of plants with all leaves infected as the alternative status (Stage 1 in Figure 2). These images are available in Underlying data.¹¹ We used them for this initial training stage of RustNet, which was modified with a pre-trained ResNet-18.⁹ The testing on a published independent set of images (5,818 diseased tiles and 14,542 non-diseased tiles) that were previously labeled manually (see Underlying data¹⁰) suggested the two types of abundantly available images are beneficial to establishing essential predictability. The area under the receiver operating characteristic curve of true positive rate against false discovery rate is 0.23 compared to 0 for the random guess. Similarly, the area under the receiver operating characteristic curve of true positive rate against false positive rate is 0.64 compared to 0.5 for the random guess.

Figure 2. Development and performances of ROOSTER to detect wheat stripe rust using automatic and semi-automatic image-labeling.

A large number of images¹¹ that were automatically labeled in control status (uninfected in a) and case status (all leaves infected by wheat stripe rust in b) were used to initialize the RestNet18 (Stage 1). Images with all leaves infected were initially labeled with case status at Stage 2 and predicted by the model trained from Stage 1 (c) to navigate humans to correct labels (d). Images with plants partially infected were initially labeled as control status (e) and predicted by the model from Stage 2 to navigate humans to correct labels (f). The performances at different stages were examined by 20,360 published tile images labeled manually (5,818 diseased and 14,542 non-diseased) in an independent study (see Underlying data¹⁰). Two receiver operating characteristic (ROC) curves with false discovery rate (g) and false positive rate (h) were used to compare performances. Random guess has AUC (area under curve) of 0 for ROC of false discovery rate (g) and 0.5 for ROC of false positive rate (the diagonal dash line in h). See Underlying data¹¹ for the raw data associated with the use case.

The labels of the images with all leaves infected in Stage 1 contained errors as some tiles do not contain leaves. We made the prediction on 20 new images⁸ with all leaves infected, and manually corrected the prediction errors. The manual corrections with prediction navigation are much easier than labeling the raw images. The training (Stage 2) with the 20 new images dramatically boosted prediction accuracy (0.54 and 0.78 compared to 0.23 and 0.64 at Stage 1 for ROC of false discovery rate and false positive rate, respectively). Similarly, we made the prediction on 61 images with partial leaves infected, manually corrected the prediction errors, and added to the training data in Stage 2 to form Stage 3 training. The prediction accuracy was further improved (0.66 and 0.87 compared to 0.54 and 0.78 at Stage 2 for ROC of false discovery rate and false positive rate, respectively).

Conclusions

ROOSTER combines functions including automatic labeling, label prediction, and manual labeling in a user-friendly GUI (graphical user interface) to label and classifies images using any outsourced models in the format of PyTorch. The navigation of integrating prediction with labeling benefits human adjustment to iteratively improve predictability to use interactive deep learning to develop machine vision systems.

Data availability

Underlying data

The independent data used to test ROOSTER was sourced from Schirrmann et al., ¹⁰ see here: https://doi.org/10.3389/fpls.2021.469689). Please contact the corresponding author of this article (mschirrmann@atb-potsdam.de) to request access to the test data if interested.

Zenodo: ROOSTER underlying dataset. https://doi.org/10.5281/zenodo.7530460.¹¹

This project contains the following underlying data:

- RawImages.zip (400 input training images used to develop the model, and captured by the authors of this article).
- UseCase.zip (use case output files).

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Software availability

Software available from: https://zzlab.net/ROOSTER .

Source code available from: https://github.com/12HuYang/ROOSTER.

Archived source code at time of publication: https://doi.org/10.5281/zenodo.7320405.¹²

License: MIT

References

1. Russakovsky O, et al.: ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 2015; 115: 211–252. Publisher Full Text
2. Sahoo D, Pham Q, Lu J, et al.: Online Deep Learning: Learning Deep Neural Networks on the Fly. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, {IJCAI-18} 2660–2666 (International Joint Conferences on Artificial Intelligence Organization). 2018. Publisher Full Text
3. Adhikari B, Huttunen H: Iterative bounding box annotation for object detection. 2020 25th International Conference on Pattern Recognition (ICPR) 2021; pp. 4040–4046.
4. Chai C, Li G: Human-in-the-loop Techniques in Machine Learning. Bull. IEEE Comput. Soc. Tech. Comm. Data Eng. 2020; 37: 37–52.
5. Monarch RM: Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster;2021.
6. Le TN, Akihiro S, Ono S, et al.: Toward interactive self-annotation for video object bounding box: Recurrent self-learning and hierarchical annotation based framework. Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020. 2020; pp. 3220–3229. Publisher Full Text
7. Lutnick B, et al.: An integrated iterative annotation technique for easing neural network training in medical image analysis. Nat. Mach. Intell. 2019; 1: 112–119. PubMed Abstract | Publisher Full Text | Free Full Text
8. Tang Z, et al.: Affordable High Throughput Field Detection of Wheat Stripe Rust Using Deep Learning with Semi-Automated Image Labeling. Preprints. 2022; 2022: 2022040177. Publisher Full Text
9. He K, Zhang X, Ren S, et al.: Deep Residual Learning for Image Recognition. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2015; 2016-Decem: 770–778.
10. Schirrmann M, Landwehr N, Giebel A, et al.: Early Detection of Stripe Rust in Winter Wheat Using Deep Residual Neural Networks. Front. Plant Sci. 2021; 475.
11. Tang Z, Hu Y, Zhang Z:ROOSTER underlying dataset (1.0). [Dataset]. Zenodo. 2023. Publisher Full Text
12. Hu Y, Zhou T:12HuYang/Rooster: ROOSTER (v1.0) [Software]. Zenodo. 2022. Publisher Full Text

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 06 Feb 2023

Author details Author details

¹ Department of Crop and Soil Sciences, Washington State University, Pullman, Washington, 99163, USA

Zhou Tang
Roles: Formal Analysis, Investigation, Validation, Writing – Review & Editing

Yang Hu
Roles: Methodology, Software, Visualization, Writing – Review & Editing

Zhiwu Zhang
Roles: Conceptualization, Funding Acquisition, Supervision, Writing – Original Draft Preparation

Competing interests

No competing interests were disclosed.

Grant information

This project was partially supported by United States Department of Agriculture (USDA) National Institute of Food and Agriculture (NIFA) (Awards 2018-70005-28792, 2019-67013-29171 and 2020-67021-32460), and the Washington Grain Commission (Endowment, 126593 and 134574), which were assigned to Zhiwu Zhang.

Article Versions (1)

version 1

Published: 06 Feb 2023, 12:137

https://doi.org/10.12688/f1000research.127953.1

Copyright

© 2023 Tang Z et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

0

SEE MORE DETAILS

CITE

how to cite this article

Tang Z, Hu Y and Zhang Z. ROOSTER: An image labeler and classifier through interactive recurrent annotation [version 1; peer review: 1 approved with reservations]. F1000Research 2023, 12:137 (https://doi.org/10.12688/f1000research.127953.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 06 Feb 2023

Views

8

Reviewer Report 16 May 2023

Michael Schirrmann, Leibniz Institute for Agricultural Engineering and Bioeconomy (ATB), Potsdam, Germany

Approved with Reservations

https://doi.org/10.5256/f1000research.140499.r171116

Summary:
The authors introduce "Rooster," a software designed to assist in human data labeling for training deep learning image classification models. Rooster features an intuitive graphical user interface (GUI) that includes a selection tool. This tool utilizes a grid ... Continue reading

Summary:
The authors introduce "Rooster," a software designed to assist in human data labeling for training deep learning image classification models. Rooster features an intuitive graphical user interface (GUI) that includes a selection tool. This tool utilizes a grid overlaid on the image, allowing users to easily select tiles corresponding to specific content, such as distinguishing between diseased and non-diseased areas. Users can load an existing deep learning model to pre-select tiles based on its predictions. The selected tiles can be confirmed or modified as needed, enabling an iterative optimization process for training the model effectively.

The manuscript provides a straightforward description of the software with a case study. Software and source code are provided as well as the data to follow the use case. The software could help especially in specific fields were data is sparse such as in many agricultural related tasks for improving image classification models. Before publication, some information needs to be added and/or limitations discussed.

Regarding limitation please discuss the following questions:

Can the tool provide also multi-class training data and optimize multi-class models? Or is it restricted to binary models?

Can the system also be extended for object detection models or is it clearly restricted to classification models?
How does Rooster compete with other labeling software already available?

Please add or change the following in the manuscript:

How is the testing of the model achieved internally for model optimization? Did you split data in training and test set, e.g., is it possible to upload specific test data, or do you perform cross validation?

In the use case, training data needs to be shortly described. In Figure 2, it is unclear what Marquardt data means.

Does Rooster also show the quality of the re-trained model with some performance statistics for fast assessment during the labeling update within the GUI?

What does the threshold button do shown in Figure 1 in the GUI?

Suggest to write rather “human data labeling” than “human labeling”

Is the rationale for developing the new software tool clearly explained?

Yes
Is the description of the software tool technically sound?

Yes
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Partly
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Precision Agriculture, Remote Sensing

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 06 Feb 2023

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1
Version 1 06 Feb 23	read

Michael Schirrmann, Leibniz Institute for Agricultural Engineering and Bioeconomy (ATB), Potsdam, Germany

Comments on this article

All Comments(0)

Add a comment

Sign up for content alerts

Browse by related subjects

Back to all reports

Reviewer Report

8 Views

16 May 2023 | for Version 1

Michael Schirrmann, Leibniz Institute for Agricultural Engineering and Bioeconomy (ATB), Potsdam, Germany

8 Views Cite this report Responses(0)

Approved With Reservations

Summary:
The authors introduce "Rooster," a software designed to assist in human data labeling for training deep learning image classification models. Rooster features an intuitive graphical user interface (GUI) that includes a selection tool. This tool utilizes a grid overlaid on the image, allowing users to easily select tiles corresponding to specific content, such as distinguishing between diseased and non-diseased areas. Users can load an existing deep learning model to pre-select tiles based on its predictions. The selected tiles can be confirmed or modified as needed, enabling an iterative optimization process for training the model effectively.

The manuscript provides a straightforward description of the software with a case study. Software and source code are provided as well as the data to follow the use case. The software could help especially in specific fields were data is sparse such as in many agricultural related tasks for improving image classification models. Before publication, some information needs to be added and/or limitations discussed.

Regarding limitation please discuss the following questions:

Can the tool provide also multi-class training data and optimize multi-class models? Or is it restricted to binary models?

Can the system also be extended for object detection models or is it clearly restricted to classification models?
How does Rooster compete with other labeling software already available?

Please add or change the following in the manuscript:

How is the testing of the model achieved internally for model optimization? Did you split data in training and test set, e.g., is it possible to upload specific test data, or do you perform cross validation?

In the use case, training data needs to be shortly described. In Figure 2, it is unclear what Marquardt data means.

Does Rooster also show the quality of the re-trained model with some performance statistics for fast assessment during the labeling update within the GUI?

What does the threshold button do shown in Figure 1 in the GUI?

Suggest to write rather “human data labeling” than “human labeling”

Is the rationale for developing the new software tool clearly explained?

Yes
Is the description of the software tool technically sound?

Yes
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Partly
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Precision Agriculture, Remote Sensing

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Respond to this report

Responses (0)

[1] 1. Russakovsky O, et al.: ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 2015; 115: 211–252. Publisher Full Text

[2] 2. Sahoo D, Pham Q, Lu J, et al.: Online Deep Learning: Learning Deep Neural Networks on the Fly. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, {IJCAI-18} 2660–2666 (International Joint Conferences on Artificial Intelligence Organization). 2018. Publisher Full Text

[3] 3. Adhikari B, Huttunen H: Iterative bounding box annotation for object detection. 2020 25th International Conference on Pattern Recognition (ICPR) 2021; pp. 4040–4046.

[4] 4. Chai C, Li G: Human-in-the-loop Techniques in Machine Learning. Bull. IEEE Comput. Soc. Tech. Comm. Data Eng. 2020; 37: 37–52.

[5] 5. Monarch RM: Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster;2021.

[6] 6. Le TN, Akihiro S, Ono S, et al.: Toward interactive self-annotation for video object bounding box: Recurrent self-learning and hierarchical annotation based framework. Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020. 2020; pp. 3220–3229. Publisher Full Text

[7] 7. Lutnick B, et al.: An integrated iterative annotation technique for easing neural network training in medical image analysis. Nat. Mach. Intell. 2019; 1: 112–119. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Tang Z, et al.: Affordable High Throughput Field Detection of Wheat Stripe Rust Using Deep Learning with Semi-Automated Image Labeling. Preprints. 2022; 2022: 2022040177. Publisher Full Text

[9] 9. He K, Zhang X, Ren S, et al.: Deep Residual Learning for Image Recognition. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2015; 2016-Decem: 770–778.

[10] 10. Schirrmann M, Landwehr N, Giebel A, et al.: Early Detection of Stripe Rust in Winter Wheat Using Deep Residual Neural Networks. Front. Plant Sci. 2021; 475.

[11] 11. Tang Z, Hu Y, Zhang Z:ROOSTER underlying dataset (1.0). [Dataset]. Zenodo. 2023. Publisher Full Text

[12] 12. Hu Y, Zhou T:12HuYang/Rooster: ROOSTER (v1.0) [Software]. Zenodo. 2022. Publisher Full Text

ROOSTER: An image labeler and classifier through interactive recurrent annotation

Abstract

Keywords

Introduction

Methods

Implementation

Figure 1. ROOSTER graphic user interface and its automatic and semi-automatic image labeling through classification.

Operation

Use case

Figure 2. Development and performances of ROOSTER to detect wheat stripe rust using automatic and semi-automatic image-labeling.

Conclusions

Data availability

Underlying data

Software availability

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated