ecoSound-web: an open-source, online platform for ecoacoustics

Implementation

Coding languages, libraries, and tools. ecoSound-web is a web-based application written in PHP 7²⁴, Python 2.7 and 3.10²⁵, Javascript²⁶, JQuery 3.4²⁷, Twig 2²⁸, CSS²⁹ and HTML 5³⁰. It uses Web Audio API³¹, Sox 14.4³², Lame³³, ImageMagick³⁴ and Scikit-maad 1.3.12³⁵ software for sound and image processing, a MySQL³⁶ database for organising the data (Figure 1), a RabbitMQ³⁷ queue for file processing, Plupload 1.5 as a visual file upload tool³⁸, GADM as administrative regions for the sites³⁹, JQuery UI 1.12⁴⁰, JCrop 0.9⁴¹, Bootstrap 4.3⁴², Leaflet⁴³, Timeline.js⁴⁴, Bootstrap-selected⁴⁵, Jquery.cookie⁴⁶, DataTables⁴⁷ and the Symfony 4 process component⁴⁸ for managing the scripts execution. Further Python libraries used are: Numpy⁴⁹, Pillow⁵⁰, Audiolab 0.8⁵¹, Matplotlib⁵², SciPy⁵³ and Scikit-image⁵⁴. We containerized the project using Docker⁵⁵, which spares software developers the time for installing libraries, the database, and configuring the server. This setup allows developers to run the project on their machines quickly and free of typical installation issues like library version incompatibilities.

Figure 1. MySQL database structure in ecoSound-web.

Audio visualization and playback. The core sound visualisation and playback tasks are handled by two distinct components. First, spectrogram images are generated by the Python script ‘sound visualization tool’, which was created for the discontinued ‘Pumilio’ project⁵⁶. This script generates spectrograms by computing a Fast Fourier Transform on the waveform of the audio recording, at a user-specified window size (at 128, 256, 512, 1024, 2048, or 4096). Second, sound playback and speed control use Web Audio API, a high-level application programming interface for processing and synthesizing audio in web applications. It is included in modern browsers to take advantage of the browser resources without requiring any extra media player or library in our project.

Operation

Ecoacoustic workflow. In ecoacoustics, a general workflow is currently not comprehensively defined. We therefore combine insights from different literature sources and our own experience to propose a general workflow as follows (Table 1): 1) Data management: Acoustic data need to be backed up, archived, and organized according to space, time, and other meta-data⁵⁶. 2) Signal processing: Recordings can be amplified, re-sampled, split, filtered, compressed, etc. for facilitating the workflow⁵⁷. 3) Audio navigation: Sound recordings can be visualized with spectrograms (i.e., sonograms) or waveforms, and played back. 4) Recording annotation: specific spectro-temporal regions can be interpreted and annotated with the sound source identity or comments. 5) Acoustic analysis: Spectral, temporal, and amplitudinal properties of the recordings can be measured or summarised with acoustic indices¹⁰. 6) Collaboration: The raw and secondary data can be shared with collaborators or the public. In the following we introduce ecoSound-web to enable the ecoacoustic community to follow and use parts of the acoustic workflow introduced here.

Server installation. ecoSound-web is published in a GitHub repository⁵⁸ and needs to be installed in a web server to run. Instructions and general information regarding the setup for developers and the production server are included in the README file on GitHub. The ecoSound-web installation for local development (in the developer’s machine) is facilitated by a Docker setup. We provide a set of Docker configuration files that can also aid the server installation, but the final setup should be carried out by the server administrator (or devOps engineer) of the institution. For server installations without Docker, a step-by-step installation guide is provided in the repository.

Access. We run an online instance of ecoSound-web⁵⁹ where the use cases described below can be reproduced. The website hosts several projects belonging to different research groups. One project hosts public reference collections (i.e., reference audio libraries) for Chiroptera, Primata, and Anura, curated by project managers. Soundscape projects can be created per request. Users can access ecoSound-web (both the existing instance and future installations) via a desktop browser with an internet connection. ecoSound-web works with Windows, Linux, and MacOS operating systems and the most common internet browsers (Firefox, Chrome, Safari), but recordings above 96 kHz cannot be played back in Firefox due to browser limitations.

Projects and collections. ecoSound-web organises audio recordings within collections, which are grouped under projects. All projects can be accessed through the "Projects" menu, which provides a public overview. Projects and collections are managed in the administrative interface. Projects can contain public (i.e., open) and closed collections, accessible to defined users (c.f. “Users” section). Recordings can be uploaded into collections in most common audio formats³². PNG image previews of the spectrograms and MP3s (for audible sound) or OGGs (for ultrasound > 44100 Hz) of the audio files are generated after insertion into the database, while the original audio files are retained on the server (a download feature is planned). Audio recordings can have custom names to hide the original information present within file names. Collections’ geographic locations are shown on Leaflet-based maps with an OpenStreetMap base layer (Figure 2A); the recordings listed in the collection are dynamically filtered and clustered based on the current map extent. Collections can be displayed as 1) a list view with larger spectrograms, descriptive data, and a simple audio player, which is particularly suitable for call libraries (Figure 2B); 2) a simple gallery view displaying spectrogram thumbnails (Figure 2C); 3) a timeline view where recordings are ordered by sites on the Y axis against a navigable time axis on X (Figure 2D).

Figure 2.

The gallery (C), list (B), and timeline (D) views for recording collections in ecoSound-web, along with the interactive site maps (A).

Users. ecoSound-web has two registered user classes with differing privileges: normal users and administrators (Figure 3B). Administrators have privileges for creating, accessing, editing, and deleting projects, collections, recordings, tags, and users. They can transform users into administrators, or give management privileges to normal users for specific projects so they can act as project managers. Project managers have privileges for creating, accessing, editing, and deleting collections, recordings, tags, and users belonging to their projects. They can give tag view and review privileges to normal users. Normal users have privileges for creating, accessing, editing, and deleting their tags belonging to their collections. If applicable, they can view and review tags of other users as determined by their privileges, and thus act as peer-reviewers.

Figure 3. The administrative interface for project managers and administrators.

The recording management tab (A), and the user privileges window (B).

Spectrogram player. Recordings can be opened in the spectrogram player (Figure 4). Spectrograms are visualisations of sound where sound amplitude is shown in color or greyscale tones, time is shown on the X axis, and frequency is displayed on the Y axis. Audio channels can be displayed separately. The current spectrogram image and the compressed audio recording (MP3 for audible sound, OGG for ultrasound) can be downloaded. The spectrogram player offers various functionalities for tagging sounds: it is possible to play back sound (at speeds between 0.05 and 1), filter frequencies outside of the current spectrogram view, navigate the spectrogram (zooming per selection, shifting frame sideways, standardized playback at specific display densities), annotate spectrogram regions (creating tags per selection, assigning them to phonies, sound types or soniferous animal species, reviewing and hiding them), label recordings with pre-defined and custom labels, and perform sound analysis (using alpha acoustic indices). Spectrograms are generated after every navigation command, and audio is downloaded on-demand for playback.

Figure 4.

ecoSound-web spectrogram player. Numbers correspond to the following sections: 1: Project/collection/sound name. 2: annotating (i.e., tagging) sounds. 3: hiding/showing tags. 4: panning spectrogram left and right 5: playback mode (zoom to standardised display density). 6: audio channel selection. 7: overview spectrogram, red rectangle shows current view. 8: playback speed slider. 9: playback/pause and stop, time position. 10: playback cursor. 11: time and frequency coordinates of current view or selection. 12: two analysis buttons: copying time and frequency coordinates, exporting frequency of maximum energy. 13: zooming into current selection. 14: continuous playback. 15: frequency filter. 16: utilities menu, containing: image and audio download, FFT window size setting, file info. 17: tags of different users shown with different colors; reviewed tags with solid border, not yet reviewed tags with dashed border; tags without detection distance with orange shading. 18: tag ID and species/sound type appear on click, with buttons for editing, zooming, and estimating distance. 19: assigning pre-set or custom label to recording. 20: computing alpha acoustic indices¹⁰.

Bird community analysis (Manual annotation and peer-review with different user privileges)

Soundscape recordings can be annotated manually for bird vocalisations (or any other sound-producing organisms), and their annotations peer-reviewed by expert ornithologists, as exemplified in the collection "Upland plots dry season 2013". Users can scan recordings visually and aurally using the built-in reading mode, which zooms into the recording to display 15 pixels per second over the entire frequency range (additionally, custom display densities can be set), and enables continuous playback between spectrogram frames. For stereo recordings, the left or right channel can be selected for visually checking vocalisations that may be audible on another channel than the one currently visible. All avian species can be tagged/annotated based on rectangular spectrogram selections along the frequency and time axes. Users can choose species from the integrated species list based on the IUCN Red List taxonomy, and links to Xeno-canto and Google image searches to help the user with identification (Figure 5). Unclear identifications can be marked as uncertain and additional comments be given. Tags can be designated as reference recordings to be included into future reference collections. Tags can be zoomed into and sharing links can be generated and copied to the clipboard. Any current audio portion can be downloaded as a spectrogram or audio file for sharing with collaborators. Detection distances are estimated by using the distance estimation function (Figure 4) that enables full-spectrum viewing and playback of the tags based on a spectrogram of the first 30 s of the tag. Additional audio recordings of test tones emitted at known distances are required to help human listeners estimate detection distances in an unbiased way⁶⁰.

Figure 5. The tag editing window.

The phony and sound type selection is remembered for faster tagging. Species, sound distance, individuals, and animal sound type fields are only shown when “biophony” is selected. Sound distance estimation is greyed out because values can only be entered with the dedicated function or declared as not estimable. Green sharing button: copies a URL to the clipboard to open the exact spectrogram region. The entire right pane is only visible to users with reviewing privileges.

Acoustic recordings can be verified and validated on multiple levels to produce accurate datasets⁶¹. In ecoSound-web, tags can be peer-reviewed to validate species identifications and auxiliary data (e.g., distance). Users with reviewing privileges can either accept species identifications, revise them by suggesting other species, mark them as uncertain, or reject them by marking the annotation for deletion (Figure 5). Administrators can also check the listening durations of each user for each recording to verify that all recordings have been listened to in entirety, and to extract a measure of the sampling intensity. Finally, it is possible to train annotators - after granting them tag viewing privileges - with example annotations of other users. Subsequently, their performance in comparison to already annotated recordings, after revoking tag viewing privileges, can be tested. After the validity checks have been run, users can export the tag data through the administrative interface as a CSV file for further statistical analysis.

Bat community analysis (working with ultrasound and using the Chiroptera reference collection)

Ultrasonic soundscape recordings can be similarly analysed for bat vocalisations, as shown in the collection "Bat point counts and automated recordings 2019”. However, bat call annotation and analysis present specific challenges. First, bat calls are very short and rapid in succession (units or tens of milliseconds), which is why ecoSound-web generates new spectrograms after zooming, based on precise spectrogram selections. Frequency filtering is enabled by default, so that users hear only what they see on the spectrogram, but can also be disabled. The Fast Fourier Transform (FFT) window size can be set for all recordings accessible to the user (in the administrative interface) and for each recording (Figure 4) to visualise bat calls better by choosing the ideal trade-off between the frequency and time resolution of the spectrogram. Finally, as ultrasound is not audible, users can adjust playback frequency (continuously between 0.05 and 1) with the playback speed slider to make ultrasound calls audible.

Bat species identification is challenging, as calls from different species can be similar. Additionally to species, custom tag identities can be used for bat sonotypes (i.e., bat call types with similar characteristics across different bat species)⁶². Exact measurement of bat call features usually determines the assignment of bat calls to specific species. Using the clipboard button (Figure 4), users can copy the frequency and time coordinates of currently-selected bat calls to derive the start and end frequency, as well as call and call interval duration. Additionally, a dedicated button computes the frequency of maximal energy of the current spectrogram, a metric used for species identification. For species that have taxonomically unequivocal calls, users can refer to the reference collection “Chiroptera” to corroborate their identifications. As manual distance estimation of bat calls is impractical due to their mobility and the fact that humans cannot intuitively estimate distances of usually inaudible sounds, tags’ distances can be marked as not estimable.

Primate bioacoustics (working with reference sound libraries)

Reference calls, i.e., example recordings of a animals’ vocalisations, can accelerate the detection of the call (visually or algorithmically) and facilitate the verification of the species identities found in soundscape recordings. Large reference call libraries already exist for birds (Xeno-Canto) and bats (ChiroVox) but are lacking for many other sound-producing animal groups. Available calls from more general libraries such as tierstimmenarchiv ("tierstimmenarchiv") contain mostly recordings of captive animals or animals with unknown geolocations, resulting in unclear taxonomies. For primates, acoustic communication has been studied in detail⁶³. However, the potential of passive acoustic monitoring has only recently been acknowledged⁶⁴ and applied to analyze, e.g., individual caller identity in orangutans⁶⁵, occupancy in chimps and gibbons^66,67 or density in fork-marked lemurs⁶⁸, and reference calls have yet to be openly published.

Primate call repertoires range from 7–8 call types in ancestral primates⁶⁹ to more than 30 individual call types in bonobos⁷⁰. Many primate vocalizations transmit indexical cues - specific call signatures linked to individuality, sex, population, or species – and they are distributed over a wide range of frequencies extending in the ultrasound field for some basal primates⁶⁹. This diversity of behavior underlines the importance of their vocalizations⁷¹. Although most primate call types are probably used in social contexts over relatively short distances, there is extensive evidence for loud, long-distance calls (several hundreds of meters), that are usually used for intergroup spacing, territorial defense, alarm situations, or as mate advertisement calls^72,73. This provides additional arguments for analysing soundscapes, which can record calls of primates over large areas, to improve future primate population monitoring. However, the lack of publicly available primate reference call libraries slows down its development. Therefore, we initiated the first public primate reference call library based on georeferenced field recordings and annotated vocalisations. Vocalisations are classified into 13 behavioral contexts, such as affiliative, agonistic, contact, or alarm call (see online Guide). The collection is shown in the public “Reference collection Primata”. DOIs of the respective publication can be assigned to the reference recordings, and Creative Commons licences be chosen to describe usage rights. Distance estimation and collaborative tagging can be used as described above.

Analysing soundscapes holistically (phonies and acoustic indices)

Soundscapes contain sounds of organismal, human, and geophysical origin³, and their acoustic diversity correlates with biodiversity⁷⁴. ecoSound-web allows the annotation of sounds with these three different phonies: biophony (sounds of organismal origin), anthropophony (sounds of human origin), and geophony (sounds of geophysical origin), as well as an “unknown” category. Phony annotation is exemplified by the “Demonstration soundscapes” collection of the Worldwide Soundscapes project on ecoSound-web. Phonies are at the highest level of the sound typology, and only biophony tags allow the specification of the sound-producing species. Within phonies, sound types can be specified, but currently, no systematic typology of sound types exists, so only pre-defined and custom labels are available. In addition, any currently generated audio portion and channel can be analysed with alpha acoustic indices provided by the integrated python package scikit-maad¹⁰. Parameters can be input for each function, or left at their default values. Results can be saved by each user and downloaded as a CSV file from the administrative interface (Figure 3A) for further analysis.

Future development

We are continuously expanding the functionality of ecoSound-web. Open-source code is a requirement for future development and maintenance. However, it is not a guarantee for a sustainable project either, as some of the open-source automated sound classification tools compiled by Priyadarshani et al. in 2018⁵ are currently discontinued. In the near future, we will implement the following functions:

In ecoSound-web, we implement best coding practices and use development tools, like Docker, to facilitate developers’ work and help them engage in collaboration. We welcome new collaborators to support the project development who could become co-authors on subsequent versions of this article.