ecoSound-web: an open-source, online platform for ecoacoustics

Implementation

Coding languages, libraries, and tools. ecoSound-web is a web-based application written in PHP 7²³, Python 2.7 and 3.10²⁴, Javascript²⁵, JQuery 3.4²⁶, Twig 2²⁷, CSS²⁸ and HTML 5²⁹. It uses Web Audio API³⁰, Sox 14.4³¹, Lame³², ImageMagick³³ and Scikit-maad 1.3.12³⁴ software for sound and image processing, getID3 for audio metadata processing³⁵, a MySQL³⁶ database for organising the data (Figure 1), a RabbitMQ³⁷ queue for file processing, Plupload 1.5 as a visual file upload tool³⁸, GADM as administrative regions for the sites³⁹, JQuery UI 1.12⁴⁰, JCrop 0.9⁴¹, Bootstrap 4.3⁴², Leaflet⁴³, Timeline.js⁴⁴, Bootstrap-selected⁴⁵, Jquery.cookie⁴⁶, DataTables⁴⁷ and the Symfony 4 process component⁴⁸ for managing the scripts execution. Further Python libraries used are: Numpy⁴⁹, Pillow⁵⁰, Audiolab 0.8⁵¹, Matplotlib⁵², SciPy⁵³, Scikit-image⁵⁴, TensorFlow⁵⁵, BirdNET-analyzer⁷ and batdetect2⁵⁶. We containerized the project using Docker⁵⁷, which spares software developers the time for installing libraries, the database, and configuring the server. This setup allows developers to run the project on their machines quickly and free of typical installation issues like library version incompatibilities.

Figure 1. MySQL database structure in ecoSound-web.

Audio visualization and playback. The core sound visualisation and playback tasks are handled by two distinct components. First, spectrogram images are generated by the Python script ‘sound visualization tool’, which was created for the discontinued ‘Pumilio’ project⁵⁸. This script generates spectrograms by computing a Fast Fourier Transform on the waveform of the audio recording, at a user-specified window size (at 128, 256, 512, 1024, 2048, or 4096). Second, sound playback and speed control use Web Audio API, a high-level application programming interface for processing and synthesizing audio in web applications. It is included in modern browsers to take advantage of the browser resources without requiring any extra media player or library in our project.

Operation

Ecoacoustic workflow. In ecoacoustics, a general workflow is currently not comprehensively defined. We therefore combine insights from different literature sources and our own experience to propose a general workflow as follows: 1) Data management: Acoustic data need to be backed up, archived, and organized according to space, time, and other meta-data⁵⁸. 2) Signal processing: Recordings can be amplified, re-sampled, split, filtered, compressed, etc. for facilitating the next workflow steps. 3) Audio navigation: Sound recordings can be visualized with spectrograms (i.e., sonograms) or waveforms, navigated in the temporal and spectral dimensions, and played back. 4) Acoustic analysis: Recordings can be annotated with the sound source identity or comments, spectral, temporal, and amplitudinal properties of the recordings can be measured or summarised with acoustic indices or other metrics⁵⁹. Note that compared to the second version of the present article, we merged the annotation step into the acoustic analysis step and removed the collaboration step, which is not specific to an ecoacoustic workflow. In the following we introduce ecoSound-web to enable the ecoacoustic community to follow and use most parts of the workflow introduced here.

Installation. The ecoSound-web code is published in a GitHub repository⁶⁰. It needs to be installed in a web server or locally to run. Instructions and general information regarding the setup for developers and the production server are included in the README file on GitHub. The ecoSound-web installation for local development (in the developer’s machine) is facilitated by a Docker setup. We provide a set of Docker configuration files that can also aid the server installation, but the final setup should be carried out by the server administrator (or devOps engineer) of the institution. For server installations without Docker, a step-by-step installation guide is provided in the repository’s Wiki pages.

Online access. We run an online instance of ecoSound-web⁶¹ where the use cases described below can be reproduced. The website hosts several projects belonging to different research groups. All projects have project managers and most projects hold data from PAM schemes; one project hosts public reference collections (i.e., reference audio libraries) for Chiroptera, Primata, and Anura. Soundscape projects and project managers can be created per request to KD. Users can access ecoSound-web (both the existing instance and future installations) via a desktop browser with an internet connection. ecoSound-web works with Windows, Linux, and MacOS operating systems and the most common internet browsers (Firefox, Chrome, Safari), but recordings above 96 kHz cannot be played back in Firefox due to browser limitations.

Projects and collections. ecoSound-web organises audio recordings within collections, which are grouped under projects. All projects can be accessed through the "Projects" menu, which provides a public overview. Projects and collections are managed in the dashboard (previously “administrative interface”). Projects can contain public (i.e., open) and closed collections accessible only to defined users (c.f. “Users” section). Recordings can be uploaded into collections in most common audio formats³¹. Users can choose to downsample audio recordings during the upload stage. PNG image previews of the spectrograms and MP3s (for audible sound) or OGGs (for ultrasound > 44100 Hz) of the audio files are generated after insertion into the database and the progress of batch insertions of files is shown in the queue tab in the dashboard. The original audio files (or resampled files, if applicable) are retained on the server. Audio recordings can have custom names to hide the original information present within file names. Collections’ geographic locations (“sites” hereafter) are shown on Leaflet-based maps with an OpenStreetMap base layer (Figure 2A); the recordings listed in the collection are dynamically filtered and clustered based on the current map extent, and the map can be filtered by ecosystem type (as defined by the IUCN Global Ecosystem Typology⁶²). Collections can be displayed as 1) a simple gallery view displaying spectrogram thumbnails (Figure 2B); 2) a list view with larger spectrograms, descriptive data, and a simple audio player, which is particularly suitable for call libraries (Figure 2C); 3) a navigable timeline view where recordings are ordered by sites on the Y axis against a time axis on X that can be zoomed and panned (Figure 2D).

Figure 2.

The interactive site maps (A) are shown on top of three available collection views for recordings in ecoSound-web: vignette galleries (B), detailed list with small audio player (C), and navigable timeline (D).

Recordings. Recordings can be managed from the dashboard (Figure 3A). The original files can be downloaded individually. Their metadata are visible and editable in a table, and the read-only audio tag information of the corresponding files are displayed there (currently supporting ID3Tag, Vorbis, RIFF formats). Recordings are assigned to sites determined by their latitude, longitude, topography (both above and below sea level), depth below the water surface (for freshwater bodies) and ecosystem type. Acoustic index or AI models can be executed on batches of recordings from the dashboard, and the progress of these “jobs” can be checked in the queue tab of the dashboard, where warnings and errors are logged, and jobs can be cancelled and cleared after execution. Meta-recordings can also be integrated in batches with a CSV upload tool. Meta-recordings consist of metadata describing when and where acoustic sampling has been conducted, and with which audio settings (sample rate, channels number, bit depth, recorder and microphone). In contrast to real audio recordings, duty cycles (e.g., 1 minute on, 9 minutes off) can be specified and displayed on the interactive timelines. Large meta-recording collections have been created to provide a global overview of PAM (i.e., Worldwide Soundscapes⁶³).

Figure 3. The administrative interface for project managers and administrators.

The recordings management tab (A), which allows to run AI models (B) or acoustic index analyses (C) on batches of recordings, whose progress can be checked under the “queue” tab of the dashboard (D). User privileges window visible to administrators and project managers (E).

Users. ecoSound-web has two registered user classes with differing privileges: normal users and administrators (Figure 3E). Administrators currently include the project lead and web developers; they have privileges for creating, accessing, editing, and deleting projects, collections, recordings, tags, reviews, and acoustic index logs, and disabling users. They can transform users into administrators, or give management privileges to normal users for specific projects so they can act as project managers. Project managers have privileges for creating, accessing, editing, and deleting collections, recordings, tags, reviews, and acoustic index logs, and disabling users belonging to their projects. They can give tag view and review privileges to normal users. Normal users without project management privileges have privileges for creating, accessing, editing, and deleting their tags belonging to their collections. If applicable, they can view and review tags of other users as determined by their privileges, and thus act as peer-reviewers (i.e., annotation validators).

Spectrogram player. Recordings can be opened in the spectrogram player (Figure 4). Spectrograms are visualisations of sound where sound amplitude is shown in color, time is shown on the X axis, and frequency is displayed linearly on the Y axis. Audio channels can be displayed separately. The current spectrogram image and the compressed audio recording (MP3 for audible sound, OGG for ultrasound) as well as the original recording can be downloaded. The spectrogram player offers various functionalities for tagging sounds: it is possible to play back sound (at speeds between 0.05 and 1), filter frequencies outside of the current spectrogram view, navigate the spectrogram (zooming per selection, shifting frame sideways, playback at specific display densities), annotate spectrogram regions (creating tags per selection, assigning them to soundscape components, sound types or soniferous animal species, reviewing and hiding them), label recordings with pre-defined and custom labels and perform sound analysis using alpha acoustic indices and AI models such as BirdNET and batdetect2. batdetect2 was chosen as it performed well in comparison with other bat call detection and classification AI models and it was easily integrated as a python-based algorithm. The execution parameters of the AI models can be set in the user dialogue and their output is saved as new annotations. Spectrograms are generated after every navigation command, and audio is downloaded on-demand for playback.

Figure 4. ecoSound-web spectrogram player.

Numbers correspond to the following sections: 1: Project/collection/sound name (navigation Breadcrumb). 2: annotate (i.e., tag) sounds. 3: hide/show tags. 4: pan spectrogram left and right 5: playback mode (zoom to pre-defined display density). 6: audio channel selection. 7: overview spectrogram, red rectangle shows current view. 8: playback speed slider. 9: playback/pause (expanded sub-menu allows to activate continuous playback) and stop buttons, time position. 10: playback cursor. 11: time and frequency coordinates of current view or selection. 12: copy time and frequency coordinates to clipboard. 13: export frequency of maximum energy to clipboard. 14: zoom into current selection. 15: frequency filter toggle. 16: tags of different users shown with different colours: reviewed tags with solid border, not yet reviewed tags with dashed border; tags without detection distance with orange shading. 17: tag ID and species/sound type appear on click, with buttons for editing, zooming, and estimating distance. 18: utilities menu, containing: download compressed recording of current view, download original entire recording, download of spectrogram image, file info, recording-level FFT window size settings. 19: assigning pre-set or custom label to recording. 20: computing alpha acoustic indices. 21: executing AI models (BirdNET and batdetect2).

Bird community analysis (Manual and automated annotation, expert peer-review)

Soundscape recordings can be annotated manually or automatically for bird vocalisations, and their annotations peer-reviewed by expert ornithologists, as exemplified in the collection "Upland plots dry season 2013". Manual annotation of any other sound-producing organism is also possible. Users can scan recordings visually and aurally using the built-in reading mode, which zooms into the recording to display 15 pixels per second over the entire frequency range (additionally, custom display densities can be set), and enables continuous playback between spectrogram frames until the recording end. For stereo recordings, the left or right channel can be selected for visually checking vocalisations that may be audible on another channel than the one currently visible. All avian species can be tagged/annotated based on rectangular spectrogram selections along the frequency and time axes. Users can choose species from the integrated species list based on the IUCN Red List taxonomy, and links to Xeno-canto and Google image searches to help the user with identification (Figure 5). Unclear identifications can be marked as uncertain and additional comments be given. Tags can be designated as reference recordings to be included into future reference collections.

Figure 5. The tag editing window.

The soundscape component (currently “phony”) and sound type selection is remembered for faster tagging. Species, sound distance, individuals, and animal sound type fields are only shown when “biophony” is selected. Sound distance estimation is greyed out because values can only be entered with the dedicated function or declared as not estimable. Green sharing button: copies a URL to the clipboard to open the exact spectrogram region. The entire right pane is only visible to users with reviewing privileges. When applicable, the AI model name is displayed after “Creator type” and the confidence score of the detection shown next to it.

Automated bird identification with BirdNET can be run on any recording (or recordings batch, from the dashboard). The spatial coordinates of recording site are automatically used for filtering BirdNET results, while other execution parameters remain at their default values unless modified by the user (Figure 3B, custom species list input not yet supported). BirdNET execution will result in the creation of 3-second tags over the entire frequency range, which can subsequently be reviewed by experts. Tags can be zoomed into and sharing links can be generated and copied to the clipboard. Any current audio portion can be downloaded as a spectrogram or audio file for sharing with collaborators. Detection distances can be estimated by using the distance estimation function (Figure 4): this will execute full-spectrum viewing and playback of the 30 first seconds of the tags; additional audio recordings of test tones emitted at known distances are required to help human listeners calibrate themselves to estimate detection distances in an unbiased way⁶⁴.

Acoustic recordings can be verified and validated at multiple levels to produce accurate datasets⁶⁵. Tags can be peer-reviewed to validate species identifications and auxiliary data (e.g., distance). Users with reviewing privileges can either accept species identifications, revise them by suggesting other species, mark them as uncertain, or reject them (Figure 5). Administrators can also check the listening durations of each user (in the MySQL database) for each recording to verify that all recordings have been listened to in entirety, and to extract a measure of the sampling intensity. Finally, it is possible to train annotators - after granting them tag viewing privileges - with example annotations of other users. Subsequently, their performance in comparison to already annotated recordings, after revoking tag viewing privileges, can be tested. After these validity checks, the tag data can be exported from the dashboard as a CSV file for further statistical analysis.

Bat community analysis (Manual and automated ultrasound annotation, Chiroptera reference collection)

Ultrasonic soundscape recordings can be analysed for bat vocalisations, as shown in the collection "Bat point counts and automated recordings 2019”. However, bat call annotation and analysis present specific challenges. First, bat calls are very short and occur in rapid succession (milliseconds). Because of this, ecoSound-web generates new spectrograms after zooming, based on the spectrogram selections. Frequency filtering is enabled by default (but can be disabled), so that users hear only what they see on the spectrogram, but it can also be disabled. The Fast Fourier Transform (FFT) window size can be set for all recordings accessible to the user (in the dashboard) and for each recording (Figure 4) to visualise bat calls better by choosing the ideal trade-off between the frequency and time resolution of the spectrogram. Finally, as ultrasound is not audible, users can adjust playback frequency (continuously between 0.05 and 1) with the playback speed slider to make ultrasound calls artificially audible (analogous to time expansion).

Bat species identification is challenging, as calls from different species can be similar. Additionally to species, custom tag identities can be used for bat sonotypes (i.e., bat call types with similar characteristics across different bat species)⁶⁶. Exact measurement of bat call features usually determines the assignment of bat calls to specific species. Using the clipboard button (Figure 4), users can copy the frequency and time coordinates of currently-selected bat calls to derive the start and end frequency, as well as call and call interval duration. Additionally, a dedicated button computes the frequency of maximal energy of the current spectrogram, a metric used for species identification (usually abbreviated “FmaxE”, Figure 4). For uncertain identifications, users can refer to reference collections such as the open “Chiroptera” collection (currently covering selected Southeast Asian species) to corroborate their identifications. As manual distance estimation of bat calls is impractical due to their mobility and the fact that humans cannot intuitively estimate distances of usually inaudible sounds, tags’ distances can be marked as “not estimable”. For automated identification of UK bat species, batdetect2 can be run on any recording (or recordings batch, from the dashboard). The detection threshold stays at its default value unless changed by the user. Upon execution, batdetect2 generates precise tags covering each individually-detected call in the frequency and time dimensions, which can then be reviewed by experts.

Primate bioacoustics (working with reference sound libraries)

Reference calls, i.e., example recordings of a animal vocalisations, can accelerate the detection of the call (visually or using AI) and facilitate the verification of the species identities found in soundscape recordings. Large reference call libraries already exist for birds (Xeno-Canto) and bats (ChiroVox) but are lacking for many other sound-producing animal groups. Available calls from more general libraries such as tierstimmenarchiv ("tierstimmenarchiv") contain mostly recordings of captive animals or animals with unknown geographic locations, resulting in unclear taxonomies. For primates, acoustic communication has been studied in detail⁶⁷. However, the potential of PAM has only recently been acknowledged⁶⁸ and applied to analyze, e.g., individual caller identity in orangutans⁶⁹, occupancy in chimps and gibbons or howler monkeys^70–72 or density in fork-marked lemurs⁷³, and reference calls have yet to be openly published.

Primate call repertoires range from 7–8 call types in ancestral primates⁷⁴ to more than 30 individual call types in bonobos⁷⁵. Many primate vocalizations transmit indexical cues - specific call signatures linked to individuality, sex, population, or species – and they are distributed over a wide range of frequencies extending in the ultrasound field for some basal primates⁷⁴. This diversity of behaviour underlines the importance of their vocalizations⁷⁶. Although most primate call types are probably used in social contexts over relatively short distances, there is extensive evidence for loud, long-distance calls (several hundreds of meters), that are usually used for intergroup spacing, territorial defence, alarm situations, or as mate advertisement calls^77,78. This provides additional arguments for analysing soundscapes, which can record calls of primates over large areas, to improve future primate population monitoring. Therefore, we initiated the first public primate reference call library based on georeferenced field recordings and annotated vocalisations. Each recording is assigned to a selectable Creative Commons license and attributed to its recordist. The collection currently contains 76 reference calls of 32 primate species from Indonesia, Myanmar, Madagascar and Peru. Vocalisations are classified into 13 behavioural contexts, such as affiliative, agonistic, contact, or alarm call (see online Wiki). The collection is shown in the public “Reference collection Primata” and is open for contributions. DOIs of the respective publication can be assigned to the reference recordings. Distance estimation and collaborative tagging can be used as described above.

Soundscape analyses (soundscape components and acoustic indices)

Beyond particular animal species, soundscapes have components that inform us about organismal, human, and geophysical events⁴, and their diversity can be measured with acoustic indices¹¹. ecoSound-web hosts the Worldwide Soundscapes project⁶³, a global inventory of PAM conducted in all IUCN realms (i.e., terrestrial, marine, freshwater, and subterranean). Recording sites were assigned to specific ecosystem types, and the interactive sampling site maps can be dynamically filtered to show particular ecosystem selections on three levels (realm, biome, functional group), thereby allowing to identify sampling gaps. Recordings of the project’s demonstration collection, containing samples from different realms, were annotated with three different soundscape components: biophony (sounds of organismal origin), anthropophony (sounds of human origin), and geophony (sounds of geophysical origin) and validated by their recordists using the tag peer-review functionality. The soundscape occupancy (i.e., the occupied proportion of the spectro-temporal space) was thereby analysed after exporting the validated tags’ data as CSV in order to analyse macroecological biophony trends. Soundscape components are the highest level of the sound typology, and sound types can be specified within, using custom labels. However, no systematic typology of sound types currently exists. Furthermore, any currently generated audio portion and channel, and any batch of recordings selected in the dashboard, can be analysed with alpha acoustic indices — which in some cases correlates with biodiversity¹¹ — provided by the integrated python package scikit-maad⁵⁹. Parameters can be input for each function or left at their default values and multiple indices can be chosen for batch analysis jobs (Figure 3C). Results can be saved by each user and downloaded as a CSV file from the dashboard’s “index logs” tab (Figure 3D) for further analysis of the acoustic diversity.

Future development

We are continuously expanding the functionality of ecoSound-web. Open-source code is a requirement for future development and maintenance, but it is not a guarantee either, so we welcome new collaborators to support the project development who could become co-authors on subsequent versions of this article. In the future, we plan to implement the following functions: