Annotation and curation of human genomic variations: an ELIXIR Implementation Study [version 1; peer review: 1 approved with reservations]

Background: ELIXIR is an intergovernmental organization, primarily based around European countries, established to host life science resources, including databases, software tools, training material and cloud storage for the scientific community under a single infrastructure. Methods: In 2018, ELIXIR commissioned an international survey on the usage of databases and tools for annotating and curating human genomic variants with the aim of improving ELIXIR resources. The 27-question survey was made available on-line between September and December 2018 to rank the importance and explore the usage and limitations of a wide range of databases and tools for annotating and curating human genomic variants, including resources specific for next generation sequencing, research into mitochondria and protein structure. Results: Eighteen countries participated in the survey and a total of 92 questionnaires were collected and analysed. Most respondents (89%, n=82) The authors systematically analysed the results and statistics of the questionnaire for an international survey on the usage of ELIXIR databases and tools, which are mainly devoted to annotating and curating human genomic variants. This survey would benefit both ELIXIR and researchers in helping them to identify the most suitable and robust bioinformatics resources, and thus make those resources well-maintained. I have two major concerns regarding this work. It is not clear how the 52 resources for Q4 (How important the following resources in work) 2 annotating curating human genomic variants) were selected. As not all of the resources are related with human genomic variants annotation and curation, the results or the question itself may be misleading. For example, Europe literature critical/important in the manuscript misunderstanding.


Introduction
ELIXIR is an intergovernmental organization, primarily centred in Europe, established to host resources, such as databases, software tools, training material and cloud storage for the scientific community, under a single infrastructure. ELIXIR started in 2013 and now includes 22 state members and over 220 research Institutions. By providing a one-stop-shop for the scientific community, it aims to help researchers identify the most suitable bioinformatics resources (and the appropriate training material and workshops) to deliver their research task. Moreover, ELIXIR recognizes and facilitates sharing of data and exchange of expertise between its members, with the goal of agreeing on best practice.
In 2018, ELIXIR commissioned an international survey on the usage of databases and tools for annotating and curating human genomic variants with the aim of improving ELIXIR resources. Here we present the result of the survey.

Methods
A 27-question survey (Extended data) was designed and agreed upon by several ELIXIR members from various European countries. The survey was constructed around six main themes (detailed in "Questionnaire structure") aimed at exploring the usage of ELIXIR resources and tools. Six ELIXIR nodes/partners informed the construction of the survey. Two researchers/research centres were identified by each of the six ELIXIR nodes to participate in a pilot test. The survey was modified based on the feedback received. The final questionnaire was made available on-line using the Webropol survey website between September and December 2018. Responses were uploaded to a Finland-based university server. Members of the Finland ELIXIR node accessed the responses and stored them in a .csv file for analysis. Information about participation was displayed on the survey and completion of the survey was taken as confirmation of participation.
Considering the absence of identifying information in data published here, and the non-sensitive nature of the survey, no ethical approval was sought for this study. No information presented here can be used to identify survey participants.

Participants
For each country, an ELIXIR representative was asked to recruit prospective survey respondents from academia, clinical diagnostics, clinical research, industry and the Government within their country. No additional eligibility criteria were set. Participation to the survey was advertised using several means, such as the mailing lists of universities, research centers, private companies and research institutes within each European country. Prospective participants were asked to fill the anonymized on-line questionnaire. A three-month deadline was given for completing the on-line questionnaire. Reminders to complete the survey were published in the regular newsletters at the recruited institutions. This approach did not allow calculation of the response rate.

Data collected
The survey collected two types of data: -quantitative data: this included category ranking metrics, as well as general frequency of use of the tools and databases surveyed. Whenever possible, a list of possible choices was provided for selection, to allow proper survey analysis and interpretation.
-qualitative data: this included participant comments.

Questionnaire structure
The survey was divided in six sections (the full questionnaire with the list of tools, databases and resources that were surveyed is presented in Extended Data): were only open to participants who worked or had an interest in working in these specific fields. Accordingly, respondents were asked to skip the sections that were not pertinent to their work or that of their group.
The complete list of tools and databases surveyed is presented in Extended data.

Data analysis
Data analysis was performed using RStudio (version 0.98.1062) and R (R version 3.4.4) 1 . Quantitative data are presented as absolute and relative frequency. Qualitative data were analysed per theme.

Results
A total of 92 questionnaires were available for analysis. Data were collected from 18 European countries and the Unites States (Extended data: Figure S1). Finland contributed to the majority of answers (30%, n=28), followed by the United Kingdom (11%, n=10). The large majority of respondents (89%, n=82) identified themselves as belonging to academia or a research environment (Extended data: Table S1). The large majority (51%, n=47) of respondents gave answers to the questionnaire in relation to their work and that of a small group of (<10) people. This was followed by answers given in relation to individual work (33%, n=30) and the work of a large group of (>10) people (16%, n=15). As the survey was sent to colleagues by the organisers, the responders may well not be representative of the wider community. Additionally, most of the responders were from European countries and the responses could be substantially different elsewhere, such as from the USA.
Key resources Section 2 of the survey aimed at identifying key data resources and tools used by the scientific community primarily in ELIXIR member states. Respondents were asked to rank 52 resources (listed in Extended data: Table S2). A total of 2055 responses were collected from 88% (82) responders. In total, 22 resources were considered critical/very important by more than 50% of respondents who knew the resources and considered them relevant to their work ( Figure 1). Three ELIXIR resources, Ensembl 2 , the Ensembl Variant Effect Predictor (VEP) 3 and UniProt 4 , were within the top ten. Several additional resources were listed by respondents and are presented in Extended data: Tables S3 and S4.
Data/data formats Four types of data/data formats used in human genome variation analysis (VCF/BCF, BAM, FASTQ, FASTA) were considered critical/very important by the majority of respondents (Extended data: Table S5). Additional data formats that were listed as critical/very important were: BED (cited 4 times), gVCF (cited twice), PLINK 8 , IMPUTE 9 and Hail MatrixTable. Genome-wide association studies (GWAS) and rare disease studies were listed as the most important topics and analytical operations in human genome variation analysis performed by respondents to this survey (Extended data: Table S6).

Important resource features
One of the main goals of this survey was to identify user requirements in the use of resources for human genome variation annotation and curation. A total of 777 responses on the importance of basic features of databases and tools were analysed. The majority of respondents (>83%) considered critical/very important a resource's free/low cost, open license to academia, the ability to assess the quality of its data, good user documentation and its availability in English. An easy-to-use web browser and privacy policy were deemed critical/very important by >77% of respondents (Table 1).
A total of 1173 responses were analysed on the importance of technical features in variation analysis, annotation tools and databases. Good curation of the database was the top scoring technical feature, ranked critical/very important by 92% of respondents. The following five technical features were considered critical/very important by >75% of respondents: i) fit for purpose data used by the resource, ii) the scientific coverage and comprehensiveness of the resource; iii) scalability to high-throughput analysis; iv) availability of datasets for download; and v) ability to analyse large datasets/queries ( Table 2).

Next Generation Sequencing
The next section of the survey focused on Next Generation Sequencing (NGS). 66 (72%) respondents worked with NGS and ranked 6 sequencing methods listed in Extended data: Table S7. Whole exome, panel DNA, whole genome and RNA sequencing were used by more than 60% of respondents. Methylation sequencing was the least used technique (28% of respondents) but was the technique the majority of respondents would like to use (23% of respondents). Several other techniques were listed among "Other", such as single-cell RNA sequencing, ribosome profiling, chromatin conformation techniques (3C, 4C, HiC, capture HiC), assay for transposase-accessible chromatin using sequencing (ATAC-SEQ), targeted RNA sequencing, fusion genes on cDNA, microbiome sequencing, GRO-seq.   The next section of the survey explored the use of resources related to the study of mitochondria. Only 15 (16%) respondents worked with mitochondrial DNA (mtDNA) alone or in combination with nuclear genes involved in mitochondrial functionality. Of these, 7 used NGS alone, 2 NGS and Sanger sequencing and 6 could not specify. Additional methods listed under "Other methods" were whole genome sequencing (WGS), whole exome sequencing (WES), RNA-seq and mtDNA included in WGS or WES.

Mitochondrial resources
Among the mitochondria-dedicated databases, MITOMAP 10 was ranked as critical/very important by 65% of respondents, followed by HmtDB (Human Mitochondrial Database) 11 by 44% and HmtVar (Human Mitochondrial Variants Database) 12 by 33%. With the exception of MITOMAP, MitoCarta 13 and HmtDB, more than half of the respondents were unaware of all other databases (Extended data: Table S8).
Among the six tools for mitochondria-related research that were ranked, MToolBox 14 , HaploGrep2 15 , MitoTip 16 and MitImpact2 17 were equally reported as critical/very important by 18% of respondents (Extended data: Table S9). The following mitochondrial databases/tools not included in the survey were listed among "Other" and ranked critical/very important: MitoFates 18 and an in house built annotation pipeline. mvTool Key limiting factors to the use of tools for modelling the structural consequences of variants were the lack of expertise in the area (54%, n=13/24), difficulties in using the tools (42%, n=10/24), and difficulties in interpreting the results (38%, n=9/24) (Figure 3). Lack of high throughput capability was listed among "Other limitations". The difficulty to translate protein dynamics into structural models was also cited.

ELIXIR
The last section of the questionnaire covered questions related to ELIXIR as a platform that coordinates the tools and databases surveyed in this study and the benefits that ELIXIR, as a European intergovernmental organization, offers to the scientific community. The majority of respondents (52%, n=42/92) were not aware that the tools and databases surveyed in Q25 are part of ELIXIR (Extended data: Figure S2). However, the long-term sustainability of ELIXIR core resources and the verified scientific quality of datasets were considered critical/very important by >76% (n=68) of participants who answered this question. The international standards for describing and saving data, the verified quality and maintenance of ELIXIR resources and facilities to find the right research tools were also considered critical/very important by the majority of respondents (>60%) ( Table 4). Additional challenges that were identified by respondents regarding the annotation and curation of human genetic variants were related to the quality of data: incorrect entries, conflicting annotation, limited access to expert curation and classification of variants  obtained in routine diagnostic setting and limited number genotype-phenotype annotations. An important caveat is the ascertainment bias that the survey was primarily sent to colleagues of the authors and this probably is not a representative sample of the community involved in the annotation and curation of human genomic variations.

Conclusions
This survey shows that the scientific community in ELIXIR member states considers several resources supported by ELIXIR crucial or very important Moreover, it shows that the work done by ELIXIR is greatly valued. In particular, most respondents acknowledged the importance of key features and This project contains the following extended data: -Copy of the online survey - Figure S1. Number of responses from the countries that participated in the survey - Figure S2. Results for Q25: "Before the survey, were you aware that the resources listed below are part of ELIXIR?
-Tables S1. Answers to Q2: Place of work. More than one answer was allowed per participant -

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Partly
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Big Data Integration and Analytics, Non-coding RNA Annotation and Curation I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com