F1000Research F1000Research 2046-1402 F1000 Research Limited London, UK 10.12688/f1000research.19801.3 Software Tool Article Articles bwimage: A package to describe image patterns in natural structures

[version 3; peer review: 2 approved]

 Biagolini-Jr. Carlos Conceptualization Data Curation Formal Analysis Methodology Software Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0002-3699-3337 a 1 Macedo Regina H. Conceptualization Funding Acquisition Supervision Writing – Original Draft Preparation Writing – Review & Editing 2 Programa de Pós-Graduação em Ecologia, Universidade de Brasília, Brasília, DF, 70910-900, Brazil Departamento de Zoologia, Universidade de Brasília, Brasília, DF, 70910-900, Brazil c.biagolini@gmail.com

No competing interests were disclosed.

 14 4 2020 2019 8 1168 7 4 2020 Copyright: © 2020 Biagolini-Jr. C and Macedo RH 2020 This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Currently R is the most popular software for data analyses among biologists. Here, we present bwimage, a package designed to describe patterns from black and white images. The package can be used for a wide range of applications. We implemented functions previously described in the literature to calculate parameters designed originally, but not exclusively, for vegetation structures. Additionally, we propose a new parameter: the aggregation index. We demonstrate applications for field work, providing examples that range from calculation of canopy openness, description of patterns in vertical vegetation structure, to patterns in bird nest structure. We provide advice and illustrated examples of how to produce high quality images for analyses.

 animal ecology methods field image analyses image processing vegetation patterns Coordenação de Aperfeiçoamento de Pessoal de Nível Superior Conselho Nacional de Desenvolvimento Científico e Tecnológico CB received graduate fellowships from Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) and Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Amendments from Version 2

Typos were fixed, and Dr. Roy Francis was included in the Acknowledgments section.

 Introduction

The facility to obtain high quality digital images creates the opportunity to measure natural variables using image analyses. Black and white pictures have frequently been used to understand patterns in field ecology, especially in plant biology studies 1 . However, the use of plant image analyses software is not easily extended to other biological fields for several reasons. Free programs are uncommon and paid software normally has threshold algorithms that were specifically designed for vegetation pictures 2 . Thus, a flexible method that would allow the application of such analyses to other subjects would be welcome. For example, despite the relatively well reported descriptions of bird nests and egg morphology in Del Hoyo and collaborators 3 (but see Xiao, Hu 4 ), there are no well-established approaches to estimate nest wall openness patterns.

Currently, R software 5 allows users to migrate from data processing based on combinations of different software (with the possibility of having costly licensing, software-specific files, incompatibility between operating systems and lack of updates) to a free, single cross-platform software. Here, we introduce bwimage, a package for R that can be used to analyze patterns in black and white images from natural structures. We provide data examples for applications and descriptions of routines for processing of black and white images.

 Methods Implementation

Bwimage´s analysis of images is based on the transformation from a picture (“jpeg” and “png” files are allowed) to a binary matrix ( Figure 1). For each pixel, the intensity of red, green, blue, or the average of these three channels (argument channel) is compared to a threshold (argument threshold_value). If the average intensity is less than the threshold (default is 50%) the pixel will be set as black, otherwise it will be white. Beyond RGB intensity in PNG images, the alpha channel is used to set transparent pixels, i.e . alpha channel values above the threshold (argument threshold_value; default is 50%) will set the pixel as transparent. In the data matrix, the value one represents black pixels, zero represents white pixels and NA represents transparent pixels. For high resolution files, i.e. numbers of pixels in width and height, we suggest reducing the resolution to create a smaller matrix, as this strongly reduces GPU usage and time necessary to run analyses. However, by reducing resolution, the accuracy of data description will also be lowered. Figure 2 compares different resamplings from a figure of 2500×2500 pixels. If the user is not acquainted with scale and threshold processing and/or images were captured under different light conditions, we recommend the scale and application of threshold algorithms in a native image editor software, such as GIMP 6 , and subsequent usage of the resulting images with the bwimage package.

General approach for image analysis in the bwimage package.

A) An image of a natural structure is obtained with digital photography; here we used an image from a canopy. B) The image is converted into a binary matrix, functions threshold_color (to a single image) or threshold_image_list (for two or more images). In the data matrix the value one represents black pixels, zero represents white pixels and NA represents transparent pixels.

 Comparison of different resamplings from a figure of 2500×2500 pixels.

A) Original image. B) Black and white conversion of the original image by GIMP software, i.e. all pixels converted to either black or white in image. C– I) conversion of the original image to binary matrices of 10×10, 50×50, 75×75, 100×100, 250×250, and 500×500, respectively.

Several metrics can be performed with the functions presented in Table 1. We implemented functions to calculate parameters designed originally, but not exclusively, for vegetation structures (described by Zehm et al. 2003) and propose a new parameter: the aggregation index. The aggregation index is a standardized estimation of the average proportion of same-color pixels around each image pixel. First, the proportion of same-color neighboring pixels (SCNP) is calculated (marginal lines and columns are excluded). Next, the SCNP for all pixels are averaged; then, given the proportion of black and white pixels, number of pixels in height and width, and location of transparent pixels (when present), the maximum and minimum possible aggregation indexes are calculated. Finally, the observed aggregation is standardized to a scale where the minimum possible value is set at zero and the maximum value is set at one ( Figure 3).

Main functions of the bwimage package.
Functions Description
Image processes
threshold_color Convert an image into a matrix
threshold_image_list Convert several images into a list of matrices
image_information Summary of image information
compress Map data from square to circular image matrix
stretch Map data from circular to square image matrix
Image analyses
denseness_total Proportion of black pixels in relation to all pixels
denseness_row Proportion of black pixels in rows subsets
denseness_column Proportion of black pixels in column subsets
hole_row Description of white pixel continuous sequences in rows
hole_columm Description of white pixel continuous sequences in columns
hole_section_data Summary information of holes of a given color in a given section
light_gap Left and right distances from first black pixel to image edge
heigh_maximum Higher black pixel in all image
altitudinal_profile Description of higher black pixel in image sections
heigh_propotion Height below which a given proportion of vegetation denseness occurs
topline Line running along the crest of highest black pixel
aggregation_index Pixel aggregation estimator
Demonstration of aggregation index calculation.

Images represent photos of 6×6 pixel, with 50% black and 50% white pixels. For the aggregation index calculation, first, the proportion of same-color neighboring pixels (SCNP) is calculated (marginal lines and columns are excluded). Then, SCNP are averaged. In these examples, the average SCNP is 0.75 ( A), 0.5 ( B), 0 ( C) and 0.875 ( D). Next, the maximum and minimum possible aggregation indexes are calculated. In images with 50% black and white pixels, the minimum aggregation will be pixels distributed in chessboard style ( C), and the highest aggregation will be the aggregation of all same color pixels on each image side ( D). Thus, by scaling the aggregation by the minimum and maximum possible aggregation, the final aggregation index is: 0.857 ( A), 0.571 ( B), 0 ( C) and 1 ( D).

 Operation

Bwimage is written in the R programming language 5 , and can be run on Windows, Mac OS X, and Linux systems. The package is available at the CRAN repository, and the development releases are available at Github (see Software availability) 7 . The bwimage CRAN page documents package dependencies. Input images must be in one of the following formats PNG, JPG, or JPEG.

 Use cases

Canopy openness is one of the most essential ecological parameters for a field ecologist. In the bwimage package, canopy openness can be calculated based on a single picture. To illustrate, we demonstrate below how to analyze a canopy image with the bwimage package. The photo was taken with a digital camera placed in the ground, perpendicular to the ground. Canopy closure can be calculated by estimating the total amount of vegetation in the canopy. Canopy openness is equal to one minus the canopy closure. For this example, we used the original image from Figure 1. The original image file is provided as Underlying data 8 .

canopy_matrix<- threshold_color("canopy.JPG",compress_method="proportional",compress_rate=0.1) 1-denseness_total(canopy_matrix) [1] 0.1297333

Several metrics to describe vertical vegetation complexity can be performed by the bwimage package (see Table 1). Here we provide examples based on an image ( Figure 2A) from a vegetation plot of 30×100cm 1. The original image file is provided as Underlying data 9 . On the 100cm side of this plot we placed a panel of 100×100 cm, covered with white cloth, and perpendicular to the ground. A plastic canvas of 50x100cm was used to cover the vegetation along a narrow strip in front of a camera positioned on a tripod at a height of 55 cm. A photograph of the portion of standing vegetation against the white cloth was taken.

vegetation_matrix<-threshold_color(bush) denseness_total(vegetation_matrix,height_size = 100, width_size = 100) [1] 0.115248 topline(vegetation_matrix) topline 785.6 heigh_propotion_test(vegetation_matrix,proportion=0.75,height_size=100) Height below which 0.75 of the vegetation denseness is located 31.2 aggregation_index(vegetation_matrix) adjusted_aggregation non_adjusted_aggregation 0.8512586 0.9634373

Variation in eggs and nest morphology provide relevant information concerning bird life history that has frequently been used to answer ecological 10– 12 and evolutionary questions 13– 15 . Here we analyze examples that address the quantification nest wall openness and the aggregation of nest wall holes, using a nest of the blue-black grasssquit ( Volatinia jacarina) deposited in the museum collection Coleção Ornitológica Marcelo Bagno, at Universidade de Brasília (register number COMB-N682). Figure 4 describes how to produce a high-quality image to describe patterns in bird nest wall openness. The original image file used is provided as Underlying data 16 .

NestWall_matrix<-threshold_color("NestWall.png", filetype="png",compress_method="width_fixed",target_width=300) denseness_total(NestWall_matrix) [1] 0.7612406 aggregation_index(NestWall_matrix) adjusted_aggregation non_adjusted_aggregation 0.9007821 0.9502800

Description of how to produce an image to evaluate patterns in bird nest wall openness.

The following example used a blue-black grasssquit ( Volatinia jacarina) nest deposited in the museum collection Coleção Ornitológica Marcelo Bagno, at Universidade de Brasília (catalog number COMB-N682). A) A photo of the nest was taken with a white styrofoam ball (50mm in diameter) placed inside the nest chamber. B) In GIMP software (version 2.10.10), we added alpha channel (to allow transparent pixels), and used the Ellipse select tool to select contrast of the white ball and nest wall, and then erased all background elements. C) To ensure that shadows from nest wall did not influence white ball detection, we also performed the threshold processing in GIMP. To preserve the transparent pixels from background elements, the final image must be saved as a PNG file.

 Conclusions

The bwimage package provides accessible and simple methods for ecologists and field researchers to describe patterns from black and white digital images. It is a flexible method that allows the application of image analyses to an exceptionally broad range of research subjects. Bwimage´s analysis is based on a simple computational routine based on the transformation of a picture (“jpeg” or “png” files) into a binary matrix, followed by the analysis itself. Several metrics can be calculated by the bwimage package. We implemented functions previously described in the literature, and additionally, we proposed a new parameter: the aggregation index, which generates a standardized estimate of the average proportion of same-color pixels around each image pixel. The application of this methods is exceptionally broad.

 Data availability Underlying data

Figshare: Canopy of Royal poinciana. https://doi.org/10.6084/m9.figshare.8429117.v2 8

Figshare: Image from a vegetation plot of 30x100cm. https://doi.org/10.6084/m9.figshare.8429882.v2 9

Figshare: Blue-black grassquit (Volatinia jacarina) nest. https://doi.org/10.6084/m9.figshare.8432018.v1 16

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

 Software availability

Software available from: https://CRAN.R-project.org/package=bwimage

Source code available from: https://github.com/biagolini/bwimage

Archived source code at time of publication: https://doi.org/10.5281/zenodo.3266299 7

License: GPL-3

 Acknowledgments

We are grateful to Universidade de Brasília for logistic support and Marcelo Antônio de Assis Silva for technical advice on how to obtain a high-quality image of the bird nest used in the manuscript. We also acknowledge Dr. Francesco Chianucci and Dr. Roy Francis for comments in a previous version of the manuscript.

 Zehm A Nobis M Schwabe A : Multiparameter analysis of vertical vegetation structure based on digital image processing. Flora. 2003;198(2):142–60. 10.1078/0367-2530-00086 Nobis M Hunziker U : Automatic thresholding for hemispherical canopy-photographs based on edge detection. Agr Forest Meteorol. 2005;128(3–4):243–50. 10.1016/j.agrformet.2004.10.002 Del Hoyo J Elliott A Sargatal J : Handbook of the Birds of the World Alive. Barcelona: Lynx Edicions;2018. Xiao H Hu Y Lang Z : How much do we know about the breeding biology of bird species in the world? J Avian Biol. 2016;48(4):513–8. 10.5061/dryad.m7ph3 R Core Team: R: a language and environment for statistical computing.2019. Kimball S Mattis P Natterer M : GIMP Image Manipulation Program.2019. Reference Source Biagolini-Jr C : biagolini/bwimage: Describe Image Patterns in Natural Structures (Version 1.01). Zenodo. 2019. http://www.doi.org/10.5281/zenodo.3266299 Biagolini-Jr C : Canopy of Royal poinciana.2019. http://www.doi.org/10.6084/m9.figshare.8429117.v2 Biagolini-Jr C : Image from a vegetation plot of 30x100cm.2019. http://www.doi.org/10.6084/m9.figshare.8429882.v2 Poláček M Bartíková M Hoi H : Intraclutch eggshell colour variation in birds: are females able to identify their eggs individually? PeerJ. 2017;5:e3707. 28875071 10.7717/peerj.3707 5581944 Gómez J Ramo C Troscianko J : Individual egg camouflage is influenced by microhabitat selection and use of nest materials in ground-nesting birds. Behav Ecol Sociobiol. 2018;72(9):142. 10.1007/s00265-018-2558-7 Wang L Cheng SJ Hsu YC : Nest-dismantling behavior of yellow-bellied prinia in mainland and island populations. Acta Ethol. 2018;21(1):35–41. 10.1007/s10211-017-0281-4 Leighton GM : Evolutionary mechanisms maintaining nest construction in avian clades. Avian Biol Res. 2016;9(1):44–51. 10.3184/175815516X14500793412915 Deeming DC : How does the bird-nest incubation unit work? Avian Biol Res. 2016;9(2):103–13. 10.3184/175815516X14567543242701 Stoddard MC Yong EH Akkaynak D : Avian egg shape: Form, function, and evolution. Science. 2017;356(6344):1249–54. 28642430 10.1126/science.aaj1945 Biagolini-Jr C : Blue-black grassquit (Volatinia jacarina) nest.2019. http://www.doi.org/10.6084/m9.figshare.8432018.v1 10.5256/f1000research.23161.r59116 Reviewer response for version 2 Francis Roy 1 2 Referee Department of Medical Biochemistry and Microbiology, Uppsala University, Uppsala, Sweden National Bioinformatics Infrastructure Sweden, Uppsala, Sweden

Competing interests: No competing interests were disclosed.

 13 2 2020 Copyright: © 2020 Francis R 2020 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve

Describing image patterns in natural structure is not my area, therefore my comments are quite general.

I have tried running the tool on two different OS and tried out most of the functions using several images. Generally, the tool and functions work as described. The documentation is reasonably well written and easy to follow.

R is quite lacking in image analysis, therefore this package could be a useful little addition.

For effectively following the workflow described here, optimal thresholding is the key. The thresholding offered with the package is too basic and the user has to rely on external tools.

To compare the resulting metrics, two images need to be exactly comparable. The zoom/crop/scaling of the image, the lighting conditions, shadow etc would most likely render the results incomparable. This is something to be handled by external tools as well.

My opinion is that image recognition and classification has to be implemented as some sort of machine learning algorithm rather than any manual thresholding for practical real world application. See relevant example 1 .

No implementation to work with large images (Sparse matrices etc). Only available option is to scale down the image and work with a smaller image. Issues with small image size is briefly discussed in the manuscript.

Summary functions would be nice. Some simple functions to visualise the results would be nice. For example just to preview the thresholded image, I used image(t(apply(img_thres,2,rev))).

Benchmarking: It's hard to say how good the results are. It would've been nice to see a comparison of this tool to other existing tools in terms of efficiency as well as accuracy. For example; How does SCNP compare to other measures of aggregation (used in this field)? It would be nice to see some sort of use case on real world dataset rather than a few isolated images.

A detailed HTML vignette showcasing all the functionality and typical use case workflows. The r-bloggers tutorial was a good start but I wouldn't consider that as a stable location. Perhaps on github along with the source?

Possible typos in function names: heigh_maximum, heigh_propotion, heigh_propotion_test, hole_columm. At least, it's consistent with Table 1 in the manuscript.

In figure 2, dimension of sub plot I is missing in the caption.

Figure 3 caption typo: Imagens.

Typo Page 8, last line of acknowledgements. acknowledg.

Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Yes

Is the rationale for developing the new software tool clearly explained?

Yes

Is the description of the software tool technically sound?

Yes

Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Yes

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Partly

Reviewer Expertise:

Molecular Biology, Bioinformatics

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

References : Automatic detection of woody vegetation in repeat landscape photographs using a convolutional neural network. Ecological Informatics .2019;50: 10.1016/j.ecoinf.2019.01.012 220-233 10.1016/j.ecoinf.2019.01.012 10.5256/f1000research.23161.r55892 Reviewer response for version 2 Chianucci Francesco 1 Referee https://orcid.org/0000-0002-5688-2060 The Council for Agricultural Research and Economics (CREA) - Research Centre for Forestry and Wood, Arezzo, Italy

Competing interests: No competing interests were disclosed.

 12 11 2019 Copyright: © 2019 Chianucci F 2019 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve

I think the author(s) has made the required correction and therefore I suggest the revised article is ready for indexing.

Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Partly

Is the rationale for developing the new software tool clearly explained?

Yes

Is the description of the software tool technically sound?

Yes

Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Partly

Reviewer Expertise:

Forestry

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

 10.5256/f1000research.21722.r51562 Reviewer response for version 1 Chianucci Francesco 1 Referee https://orcid.org/0000-0002-5688-2060 The Council for Agricultural Research and Economics (CREA) - Research Centre for Forestry and Wood, Arezzo, Italy

Competing interests: No competing interests were disclosed.

 13 8 2019 Copyright: © 2019 Chianucci F 2019 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

The topic is interesting. However, I think the package makes simple things which many other similar packages do, and omit some intermediate steps required to refine the analysis.

First, the authors should provide a list of thresholding (either single or dual) methods to binarize images. The single Otsu, the minimum algorithm, the two-corner method are some of the algorithm which should be developed. The package could also use some dependancies from other packages to implement these thresholding. For instance, the rtiff package contains the function autoThreshold which implement the Ridler-Calvard thresholding. I also suggest take a look at the caiman package which contains tool for canopy image analysis, see also below.

Second, users can also process single channel image or decide to use a single channel (for example, the blue channel is frequently used for canopy images). So the user should have flexibility on choosing the image feature to process (setting the channel, setting the threshold).

Canopy images are usually fish eye images, and therefore they require correction for lens projection. In addition, canopy openness should be weighted for zenith angle in fish eye image. Similarly, gap fraction is required for zenith ring. Thus, the packages should allow to work with fish eye (circular image), namely correct for lens distortion, setting the circular inner mask, extract information for zenith angle ranges (inner rings). I suggest to take a look in the caiman package to inherit its functions.

An interesting attribute from the packages is the identification of row or column gaps (holes), namely continuous sequences of white pixels. In my view, a very interesting output would be the identification and the labelling of all holes in the image, along with their summary statistics (e.g. number of holes, size of each hole in number of pixels, average, sd and so on). I think the package will improve strongly if authors can implement such function. This would allow the extraction of canopy attributes from images 1 .

Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

Partly

Is the rationale for developing the new software tool clearly explained?

Yes

Is the description of the software tool technically sound?

Yes

Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?

Partly

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Partly

Reviewer Expertise:

Forestry

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

References : An objective image analysis method for estimation of canopy attributes from digital cover photography. Trees .2018;32(3) : 10.1007/s00468-018-1666-3 713-723 10.1007/s00468-018-1666-3 Biagolini-Jr. Carlos

Competing interests: No competing interests were disclosed.

 16 10 2019

Dear Dr. Francesco Chianucci,

We appreciate all of your helpful comments, and we are certain that they have improved significantly the original version of the Package. In the new version (1.2) of bwimage package published in CRAN we: i) introduced two new functions (stretch and compress functions) to provide an additional tool to distort images. These functions are applications of algorithms for mapping images from circle to square, and vice versa, adapted from Lambers (2016 J of Computer Graphics Techniques); and ii) included the possibility of users processing single channel images.

We considered incorporating other packages for threshold in bwimage dependencies; however, we decided to avoid a cross-dependency of packages because it implies in a heavy load package, and brings possible compatibility issues with R updates.  Thus, we chose to create a tutorial (shared on https://www.r-bloggers.com/using-bwimge-r-package-to-describe-patterns-in-images-of-natural-structures/) to provide an overview of bwimage package and demonstrate examples of how to apply threshold algorithms from the package autothresholdr, following its application in bwimage package. A comparison of estimation of vegetation density from a bush image submitted to different thresholding algorithms is provided. Note: the current version of autothresholdr package (1.3.5) provide 17 threshold algorithms by function auto_thresh, covering a wide range of applications.

We do not mention in the previous version of this article, but bwimage version 1.0 already had a function (hole_section_data) to summarize holes statistics (i.e. number of holes, mean hole size, sd, minimum and maximum size).  This function was designed to be used inside a loop or apply-family functions. By combination of hole_section_data and loop function, users can potentially collect a summary of holes statistics. The size of each section is obtained by the hole_section function, which returns size and map of each hole. We add information about this function in manuscript table 1. We also provide an example of how to apply hole_section_data for a set of 12 images in the above-mentioned tutorial.

With my best regards.

Sincerely,

Carlos Biagolini-Jr