We initially drafted an interview guide (list of questions available as Extended data ⁸ ) and a preliminary curator persona, reflecting our initial hypotheses about curators and their work practices. Following this we selected five different curation teams. The selection criteria was based on the type of curation the group was involved with, i.e., extracting biological information from scientific literature and integrating it into a biological database, contrary to groups that process raw data submissions (such as sequencing data). Out of these, two teams were based at EMBL-EBI, the other three teams were based in Norway, Switzerland and Italy. For teams that were situated at EMBL-EBI, the interviews were held in person and for the other teams the interviews were conducted over conference calls. The sessions were conducted over a period of two months: between March and April 2018. We observed three project leaders and four curators from the selected teams. The participants belong to teams that focused on curating very specific experimental evidence primarily on proteins such as protein-protein interactions, the role of a protein in a complex, protein disruption, human protein functions and transcription factor regulation. One team is focussed on the annotation of human genes relevant to a particular disease and another one on curating publications reporting associations of genetic variants with diseases.

For each session we followed an iterative user research process ^{9, 10} (as outlined in Table 1):

Additionally, we further carried out two follow-up interviews with one project lead and one curator from the same team at EMBL-EBI to clarify particular curation tasks.

Furthermore, we conducted a stakeholder workshop ¹² with three project leads and four curators from four different curation teams to validate our main learnings from the interview sessions. As some curators took part both in the interviews and the workshop, overall we have engaged with 12 curators and team leads from seven different teams. The participants were presented with the preliminary curator persona ¹³ and a workflow outlining the curation process. The participants were invited to give us their feedback on these drafts and express their challenges or pain points as How Can We (HCW) questions ² . Their feedback was used to revise the curator persona and the curation process workflow and was consolidated into the curation experience map.

Based on the interview guide used for the observational study, we formulated questions to understand the immediate challenges. The survey consisted of 15 questions (see Extended data ⁸ ) ranging from, for instance, the section of the article the curators were most interested in; the types of biological entities curators look for; and whether it helps to know that a given article has been curated/accounted for in another database. The community survey was conducted online and was developed using Typeform. The survey was promoted via the mailing lists of various consortia, such as ELIXIR, International Society for Biocuration (ISB) and Alliance of Genome Resources. These widely known consortia provide a forum for developers, researchers and curators to streamline and standardise the maintenance of biological resources. The survey was conducted between December 2018 to January 2019.

We confirm that we have obtained consent to use data from the participants as per the Europe PMC privacy notice. The privacy notice is formulated in accordance with EMBL’s data protection framework. The consent was part of the survey form and the participants can take the survey on accepting the terms and conditions for data re-use.

Curator persona . We created a persona called Ashley (see Underlying data ⁸ ) to present the curators’ needs, sentiments, tasks and pain points from their own perspective in more detail to help us empathise with them ¹³ :

Ashley curates with precision and attention to detail, while trying to be as efficient as possible. Ashley is looking for very specific information about an experiment that the authors of a paper do not always report in a lot of detail. Ashley appreciates being able to ask a team mate when the “detective work” bears no fruit.

Apart from these “organic, informal discussions”, Ashley works independently during “triage”, “annotation” and to fill in the curation record in the Editor. During the latter stage Ashley tries to “translate from the author’s language to the curator’s language” using the appropriate identifiers and Control Vocabulary (CV) terms for species, proteins, methods and other important entities so that the curated evidence is referred to precisely and consistently and the annotations in the curation record are self explanatory outside the context of the paper.

This is cumbersome as a particular type of evidence is not always referred to in the same way and in enough detail in the literature. Moreover, the Editor is not integrated with the search and annotation tools, so Ashley spends a lot of time going back and forth between the paper and the Editor, translating the curatable text from the paper into CV terms, switching browser tabs and consulting notes from the online research and team discussions.

Curation experience map . The curation experience map in Figure 1 presents the identified pain points in the context of the main curation activities. As shown in the map, curation consists of four stages:

In a typical curation scenario, curators:

A pain point during “triage” is identifying relevant publications: The curators reported that most publications returned in a PubMed search are usually not relevant (i.e. they are false positives). Additionally, because they are often looking for very specific and at times underreported experimental evidence, some searches return very few papers or no papers at all.

Ambiguities in the paper about species, proteins, and relevant experimental methods may slow down curation significantly during “triage” and “annotation”. The curators highlighted that identifying the species as their main pain point as this task may take up to “75% of the curation effort”, and in the end may turn out to be irretrievable from the paper.

Furthermore, to get clarity on the details of an experiment, curators would look up specific references in the paper and do further research online. If this “detective work” is not successful, the curators will either not annotate the paper or will provide fewer annotations. The curators would also discuss unresolved questions such as the annotation of unusual data or what to do when there is no curatable data with their teammates during “organic, informal discussions”. As a last resort they would contact the authors directly; however, authors often do not respond to requests for clarification.

If there are no matching CV terms to annotate the paper new ones are requested. This can delay annotating the paper because the ontology staff are often different to those curating the papers and sometimes requesting new terms results in prolonged discussions between curators and ontologists.

It was observed that curators use different tools at each stage, which are not integrated with each other. During “triage” they would search PubMed for relevant publications and then look at a particular paper on the publisher’s site. Annotating a publication may involve downloading or printing the pdf version of the paper and highlighting curatable text. To fill in the curation record they use a bespoke tool which they call “the Editor”, which presents a template that needs to be filled in with molecule names and experimental context, supported by standardised identifiers, controlled vocabularies and ontologies as well as free text describing the experimental evidence. Most of the times the assertions that go in the database are not in the paper in the same words.

Summary of the pain points in curation workflows . The identified pain points were formulated as How Can We (HCW) questions as follows:

The survey received 42 respondents in total, covering a number of European countries, such as the United Kingdom, France, Italy and Switzerland. The majority of the participants identified themselves as ‘Scientific curator’ with over 5 years experience in curation. Broadly speaking respondents mainly curate peer-reviewed articles (43.6%), followed by review (25.5%) and preprint (16%) articles. Figure 2 shows their preference in the type of articles for curation.

Figure 3 shows the section of articles of interest to the curators: the majority look for method sections in articles. The other sections of importance are the figures/tables and their legends. Apart from these, the supplementary data seems to be a section of importance. Furthermore, as shown in Figure 4, the types of entities curators look for in articles were diverse with preference given mainly to: genes/protein curation and their functions, database accession numbers, experimental methods and gene mutations.

Respondents were asked if it was useful to know a given article was already curated by another database: the results indicate that it was useful (see Figure 5). A follow-up question was asked as to (if useful) why it was useful to know if an article is already annotated by another database, the majority of the responses ranged from: avoiding duplication if the curators belong to the same consortium, as a means of validation (in case of ontology terms), and consistency in annotations.

Text mining approaches are sophisticated and play a vital role in addressing big data questions, the results of which can contribute to supplying “leads” on key papers for curation. However, curators require a wide variety of very precise information. Addressing each of those specific requirements will be a complex task, but text mining systems can certainly provide underlying services based on broad commonalities in the requirements that can prove useful to curators. To this end, this effort has proved to be useful in terms of understanding the main challenges faced by curators. While the sample size of the community survey was small, when analysed in conjunction with the observational study we found significant commonalities on the work practices. For instance, in the survey when the respondents were asked for their biggest challenge while curating, the majority responses indicated finding relevant papers, and identifying specific information that includes genes and species.

Our research on curation practices so far indicates a need to better support curators on the following areas:

Zenodo: Results of user research project to understand data curation practices. https://doi.org/10.5281/zenodo.3209658 ⁸ .

This project contains the following underlying data:

Zenodo: Results of user research project to understand data curation practices. https://doi.org/10.5281/zenodo.3209658 ⁸ .

This project contains the following extended data:

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).