BioSounds: an open-source, online platform for ecoacoustics

Implementation

Coding languages, libraries, and tools. BioSounds is a web-based application written in PHP 7¹⁴, Python 2.7¹⁵, Javascript¹⁶, JQuery 3.4¹⁷, Twig 2¹⁸, CSS¹⁹ and HTML 5²⁰. It uses Web Audio API²¹, Sox 14.4²², Lame²³ and ImageMagick²⁴ software for sound and image processing, a MySQL²⁵ database for organising the data (Figure 1), a RabbitMQ²⁶ queue for file processing, Plupload 1.5 as a visual file upload tool²⁷, JQuery UI 1.12²⁸, JCrop 0.9²⁹, Bootstrap 4.3³⁰ and the Symfony 4 process component³¹ for managing the scripts execution. The Python libraries used are: Numpy³², Pillow³³ and Audiolab 0.8³⁴. We containerized the project using Docker³⁵, which spares software developers the time for installing libraries, the database, and configuring the server. This setup allows developers to run the project on their machines quickly and free of typical installation issues like library version incompatibilities.

Figure 1. MySQL database structure in BioSounds.

Audio visualization and playback. The core sound visualisation and playback tasks are handled by two distinct components. First, spectrogram images are generated by the Python script ‘sound visualization tool’, which was created for the discontinued ‘Pumilio’ project³. This script generates spectrograms by computing a Fast-Fourier Transform on the waveform of the audio recording. Second, sound playback and speed control use Web Audio API, a high-level application programming interface for processing and synthesizing audio in web applications. It is included in modern browsers to take advantage of the browser resources without requiring any extra media player or library in our project, and we plan to use it for generating spectrograms too.

Operation

Server installation. BioSounds is published in a GitHub repository³⁶ and needs to be installed in a web server to run. Instructions and general information regarding the setup for developers and the production server are included in the README file on GitHub. The BioSounds installation for local development (in the developer’s machine) is facilitated by a Docker setup. We provide a set of Docker configuration files that can also aid the server installation, but the final setup should be carried out by the server administrator (or devOps engineer) of the institution. For server installations without Docker, a step-by-step installation guide is provided in the repository.

Access. We run an online instance of BioSounds for our project SoundEFForTS³⁷, where most of the steps described in the use cases below can be reproduced. This website can host public reference collections (i.e., reference audio libraries) for prospective users, for instance for Chiroptera and Anura. Soundscape collections, due to their larger size, can be integrated up to a manageable size for projects contributing to BioSounds development.

Users can access BioSounds (both the existing instance and future installations) via a desktop browser with an internet connection. BioSounds works with Windows, Linux, and MacOS operating systems and the most common internet browsers (Firefox, Chrome, Safari).

Collections. BioSounds organises audio recordings (named “recordings” hereafter) within collections. Collections can be accessed through the "Collections" drop-down menu. Those that are part of ongoing research projects are only visible to registered users; open collections are public. Collection creation is still handled directly via the database by adding it to the table ´Collection´. Administrators can then upload recordings in most common audio formats into collections. PNG image previews of the spectrograms and MP3s of the audio file are generated after insertion into the database. Audio recordings can be given names that differ from the default file name. Collections can be shown with a gallery view (thumbnails with sound names) or a list view with larger spectrograms and a simple audio player, and comments can be inserted. There are two types of collections in BioSounds: soundscape recording collections ("soundscape collections" hereafter) and reference recording collections (i.e., reference audio libraries; named "reference collections" hereafter).

Soundscape collections contain field recordings which each encompass a range of sounds from a particular site during a particular time interval. The recordings within are displayed with the gallery view by default, which shows either mono or stereo thumbnails of their spectrograms along with the sound names and the overlaid maximum recorded sound frequency (Figure 2).

Figure 2. The default gallery view for soundscape recording collections in BioSounds.

Reference recording collections link individual recordings to identified sound sources (typically, sonant animal species). They display recordings with a list view by default; an example is shown in the public “Reference collection Anura”. Reference collections can host recordings that are needed for supporting the identification of the animals of particular taxa or regions. The spectrogram in the list view gives a rapid overview of the reference sound, and the embedded audio player can play it back directly (Figure 3). In contrast to soundscape recordings, uploaded reference recordings need to be assigned to animal species, and they can have a vocalisation type and quality rating. Reference collections can be filtered by species and rating.

Figure 3. Reference recording shown in list view.

Reference collections can be filtered by species and rating, and additional information regarding the recording is displayed. Reference recordings can be played back directly in list view.

Users. BioSounds has two registered user classes: normal users and administrators. All registered users can open all recordings inside the spectrogram player, as well as create, view, and edit their own annotations (called "tags" in BioSounds) that are linked to sound sources (mostly sonant animal species). Normal users have viewing and reviewing privileges for other users' tags that are set by administrators for single collections. Administrators can view, review, and edit all users' tags. They can also create users, set their tags' color, and define their status (normal user/administrator). Finally, administrators can upload, rename, and delete recordings.

Spectrogram player. Recordings can be opened in the spectrogram player (Figure 4). Spectrograms are visualisations of sound where sound amplitude is shown in color or grayscale tones, time is shown on the X axis, and frequency is displayed on the Y axis. The spectrogram player offers various functionalities for tagging sounds: it is possible to play back sound, filter frequencies, navigate the spectrogram, assign selections to animal species (or other sound sources), and perform basic sound analysis.

Figure 4. BioSounds spectrogram player.

1: sound and collection name. 2: annotating/tagging sounds. 3: hiding/showing tags. 4: playback mode. 5: moving window left and right. 6: audio channel selection. 7: overview spectrogram, red rectangle shows current view. 8: playback speed. 9: playback/pause and stop, time position. 10: playback cursor. 11: time (s) and frequency (Hz) coordinates of current view or selection. 12: copying time and frequency coordinates. 13: zooming. 14: continuous playback. 15: frequency filter. 16: utilities: image and audio download, file info. 17: tags of different users shown with different colors. 18: reviewed tags with solid border. 19: not yet reviewed tags with dashed border. 20: tags without detection distance with orange shading. 21: tag species appears on click, with buttons for editing, zooming, and estimating distance.

Bird community analysis

Soundscape recordings can be annotated manually and reviewed by expert ornithologists, as exemplified in the collection "Upland plots dry season 2013". Users can scan recordings visually and aurally using the built-in reading mode, which zooms to a 60 s long section of the recording, including all frequencies, and enables continuous playback. All avian species can be tagged/annotated based on rectangular spectrogram selections along the frequency and time axes. Species are chosen from the integrated species list, and links to Xeno-canto and Google image searches direct the user to the selected species to support identification (Figure 5). Project-specific reference collections can also be consulted to confirm species identification. Unclear identifications can be marked as uncertain. Coordinates (in time and frequency) are saved automatically based on the boundaries of the selection. Tags can be designated as reference recordings for future inclusion into reference recording collections; comments can be inserted. Tags can be zoomed into and any of the current (filtered or unfiltered) spectrogram views (image or audio) can be downloaded for sharing with collaborators. Distances are estimated in a standardised way using a function that enables full-spectrum viewing and playback of the tags based on a spectrogram of the first 30 s of the tag. Reference audio recordings of test tones emitted at known distances are needed (see recording "Sound transmission - full spectrum" in Demo collection) to estimate detection distances in an unbiased way³⁸.

Figure 5. The tag editing window.

The right pane is only visible to users with reviewing privileges.

An advantage of automated acoustic survey data is that they can be validated on multiple levels, yielding accurate datasets³⁹. In BioSounds, it is possible to review tags for validating species identification as well as auxiliary tag data. Administrators can grant tag reviewing privileges to users other than the creator. Users with reviewing privileges can either accept species identifications, revise them by suggesting other species, or reject them by marking the annotation for deletion (Figure 5). Administrators can also check the listening durations of each user for each recording to verify that all recordings have been listened to in entirety, and to extract a measure of the sampling intensity. Finally, it is possible to train users by letting them learn from other users’ annotations after granting them viewing privileges, and thereafter, to test their performance with already annotated recordings where the annotations are invisible to the user being tested. After the validity checks have been run, administrators can export the tag data through a MySQL database administration tool like phpMyAdmin⁴⁰ for further statistical analysis.

Bat community analysis

Soundscape recordings that span the ultrasound frequency range (i.e., ultrasoundscapes) can be similarly analysed with the same functions as for the bird community analysis use case, but they present specific challenges regarding the analysis of bat calls. Most importantly, ultrasound is not audible, so that users need to use the playback speed slider to reduce the playback frequency to artificially hear the ultrasound calls. This can be tested with the annotated example recording "Sample Ultrasoundscape" that is uploaded in the "Demo" collection; any playback rate from 0.05 to 1 can be chosen. To aid in bat call visualisation, the spectrogram settings can also be adjusted by administrators to choose different FFT window sizes.

However, bat species identification is more challenging as bat calls from different species can be similar. Thus, we included bat morphospecies (to be exact, morphocall types) named with single letters from A to J into the species list, suffixed with digits to designate different call types from the same species. Exact measurement of bat call features (such as start and end frequency, frequency of maximal energy, as well as call and call interval duration) usually determines the assignment bat calls to specific species: using the clipboard button (Figure 4), users can copy the frequency and time coordinates of the current selection to the clipboard to perform basic sound analysis. The exported values can be readily pasted into spreadsheets, and bat call metrics of interest can be rapidly computed with formulae. For those species that have taxonomically unequivocal calls, the users can refer to the reference collection to corroborate their identifications. Finally, manual distance estimation of bat calls is impractical due to their mobility and the fact that we cannot intuitively estimate the distances of human-inaudible sounds, so that the tags can be marked as having not estimable distances.