Identifying ELIXIR Core Data Resources

The core mission of ELIXIR is to build a stable and sustainable infrastructure for biological information across Europe. At the heart of this are the data resources, tools and services that ELIXIR offers to the life-sciences community, providing stable and sustainable access to biological data. ELIXIR aims to ensure that these resources are available long-term and that the life-cycles of these resources are managed such that they support the scientific needs of the life-sciences, including biological research. ELIXIR Core Data Resources are defined as a set of European data resources that are of fundamental importance to the wider life-science community and the long-term preservation of biological data. They are complete collections of generic value to life-science, are considered an authority in their field with respect to one or more characteristics, and show high levels of scientific quality and service. Thus, ELIXIR Core Data Resources are of wide applicability and usage. This paper describes the structures, governance and processes that support the identification and evaluation of ELIXIR Core Data Resources. It identifies key indicators which reflect the essence of the definition of an ELIXIR Core Data Resource and support the promotion of excellence in resource development and operation. It describes the specific indicators in more detail and explains their application within ELIXIR’s sustainability strategy and science policy actions, and in capacity building, life-cycle management and technical actions. The identification process is currently being implemented and tested for the first time. The findings and outcome will be evaluated by the ELIXIR Scientific Advisory Board in March 2017. Establishing the portfolio of ELIXIR Core Data Resources and ELIXIR Services is a key priority for ELIXIR and publicly marks the transition towards a cohesive infrastructure.

This paper describes the structures, governance and processes that support the identification and evaluation of ELIXIR Core Data Resources.It identifies key indicators which reflect the essence of the definition of an ELIXIR Core Data Resource and support the promotion of excellence in resource development and operation.It describes the specific indicators in more detail and explains their application within ELIXIR's sustainability strategy and science policy actions, and in capacity building, life-cycle management and technical actions.The identification process is currently being implemented and tested for the first time.The findings and outcome will be evaluated by the ELIXIR Scientific Advisory Board in March 2017.
Establishing the portfolio of ELIXIR Core Data Resources and ELIXIR Services is a key priority for ELIXIR and publicly marks the transition towards a cohesive Establishing the portfolio of ELIXIR Core Data Resources and ELIXIR Services is a key priority for ELIXIR and publicly marks the transition towards a cohesive infrastructure.
This article is included in the ELIXIR Reports channel.

Introduction
ELIXIR is an intergovernmental organisation which builds on existing data resources and services within Europe.It follows a Hub and Nodes model, with a single Hub located in Hinxton, United Kingdom, and a growing number of Nodes located at centres of excellence throughout Europe.Governments and ministries of countries are members of the ELIXIR consortium, and the scientific community in each member country develops their national Node.
The core mission of ELIXIR is to build a stable and sustainable infrastructure for biological information across Europe.At its heart are the data resources, tools and services that ELIXIR Nodes offer to the life-science community, providing stable and sustainable access to biological data.
ELIXIR resources vary from archives, or deposition databases, which contain research data outputs such as DNA sequences, to highly dynamic knowledge bases which aggregate, process and visualize research data, often adding layers of value through manual curation by highly qualified personnel.ELIXIR aims to ensure that these resources are available long-term and that their life-cycles are managed so that they support the scientific needs of life-sciences and biological research.
Over 500 data resources exist in Europe 1 .Only a small fraction of these have institutional support and long-term funding commitments.The fact that the mid-and long-term survival of many crucial bioinformatics resources is not guaranteed threatens the foundations of academic and industrial life-science activities, and risks the loss of an immense wealth of biological and medical information, and the associated investments.
Identifying ways to assess the quality and impact of these crucial data resources will (a) promote excellence in resource development and operation to support capacity building through spreading best practice, and (b) provide a basis for technical and science policy actions required to support the long-term sustainability of the resources that form the backbone of bioinformatics infrastructure (Figure 1).
The proposal for establishing ELIXIR Services and ELIXIR Core Data Resources was put to the ELIXIR Scientific Advisory Board (SAB) in December 2014 2 .This paper describes how to put the proposal into practice and provides guidelines for the implementation of life-cycle management.
ELIXIR Nodes define, through their Node applications and Service Delivery Plans or Work Programme, a set of services and data resources that are offered to the research community, the ELIXIR Services.These resources form the backbone of the lifescience data infrastructure.
ELIXIR Core Data Resources are defined as a set of European data resources that are of fundamental importance to the wider lifescience community and the long-term preservation of biological data.They provide complete collections of generic value to lifescience, are considered an authority in their field with respect to one or more characteristics, and show high levels of scientific quality and service.Thus, ELIXIR Core Data Resources are of wide applicability and usage.
ELIXIR Core Data Resources tend to be well-known within the life-science community and are known to key stakeholders such as funders and journals.ELIXIR Core Data Resources are well maintained with a professional service delivery plan based on

Amendments from Version 1
The manuscript has been updated to provide an overview of the ELIXIR structure and a timeline for the selection and revision of Core Data Resources.Figure 3 was revised to include the role independent experts play in identifying ELIXIR core data resources.

REVISED
well-established life-cycle management processes and wellunderstood dependencies with related data resources.The ELIXIR Core Data Resources coexist with a broader range of databases with diverse motivations, often specialising in a particular scientific topic.
The ELIXIR Core Data Resources will form the focal point of technical and science policy actions to drive long-term sustainability.Transparent indicators for the ELIXIR Core Data Resources will also provide strategic intelligence on resource quality and impact, notably to policy makers and funders.
Through the ELIXIR Scientific Programme and ELIXIR-EXCELERATE grant, the infrastructure will deliver and enable a range of initiatives to support and strengthen the ELIXIR Services and ELIXIR Core Data Resources.ELIXIR Services and ELIXIR Core Data Resources will be the most widely used and outwardly visible part of ELIXIR.Establishing the portfolio of these data resources and services is the key priority for ELIXIR and publicly marks the transition towards a cohesive infrastructure.The identification process is currently being implemented and tested for the first time.The findings and outcome will be evaluated by the ELIXIR Scientific Advisory Board in March 2017.Through the establishment of the ELIXIR Services portfolio, ELIXIR also aims to support and implement best practice in resource management and bring European bioinformatics resources to the next level, building confidence among users.

Measuring the quality and impact of data resources: key indicators
In their report on the role of metrics in research assessment and management in the United Kingdom 3 , Wilsdon et al. highlight that the term 'metric' may be misunderstood.For example, the number of citations received by a publication is a citation metric, as it does not directly measure the impact of that researcher's work.
They therefore suggest that the term 'indicator' is used in contexts in which there is the potential for confusion (Table 1).An 'indicator' is defined as a measurable quantity that substitutes for something less readily measurable and is presumed to associate with it without directly measuring it.Citation counts could be used as indicators for the scientific impact of journal articles, even though scientific impacts can occur in ways that do not generate citations.We therefore use the term 'indicators' throughout.

Life-cycle management of ELIXIR Services
This section outlines the framework and stages for life-cycle management of the ELIXIR Services (Table 1).This framework will be implemented

(1) Scientific focus and quality of science
This includes the inherent scientific quality of the data and of the metadata, and its uniqueness and comprehensiveness.Also included are benchmarking against other resources, and whether the resource is an authority in its field.
A differentiation should be made between archival or deposition databases that receive and archive de novo data sets and wellstructured metadata deposited by scientists, and added-value databases or knowledge bases, which are based on the archival data and add substantial value through expert curation, annotation of metadata, sophisticated data processing and/or data integration.The curation effort and outputs linked to a resource are an important measure of its quality. (

2) Community
This category reflects the size and the measured demand of the communities that are served by the resource: web statistics, user reach, and international use.The community that is served can be the depositors, as some resources are vital for deposition and/or the end-users.The community can be identified and measured in different ways, such as access to URLs, to download servers, and through APIs, and also through the citation of data and data resources in publications.
In addition, certain resources play a foundational role to derived services and data-driven research.Their data are distributed to many other resources and/or services that rely on their existence.
The scientific context in which the resource operates should be taken into account.A resource that serves a small scientific community may not have as many users as a resource serving a broader interest, and yet it may reach 90% of the community it supports (coverage) and be crucial for the scientific work of that community.
( A viable resource has a suitable legal framework (clear terms of use, licensing, data security, ethical compliance, etc.).Open data is a critical driver for life-sciences research and therefore for ELIXIR, but the policy for data access must be considered in view of resource funding.Longevity can be gauged through institutional support, funding schemes and the duration of financial stability.Core Data Resources will have demonstrated transition through different funding sources.A strong governance structure includes an international, independent Scientific Advisory Board (SAB), which allows community input and provides permanent oversight.
(5) Impact and translational stories Impact evaluation attempts to provide a definitive answer to the question of whether the resource is meeting its objective of fulfilling a specific need of the scientific community.The translational stories relate to the role of the resource in accelerating science and are thus a very important indicator.
Impact evaluation attempts to assess whether the Resource is meeting its objective of fulfilling a specific need.In the UK, the HM Treasury's Magenta Book 4 provides guidelines for policy makers and analysts on how policies and projects should be assessed and reviewed.According to this guidance, the key characteristic of a good impact evaluation is that it recognises that most needs can be met by a range of elements, not just the project in question.To test the extent to which the Resource is responsible for meeting the need, it is necessary to estimate -usually on the basis of a statistical analysis of quantitative data -what would have happened if the Resource had not existed.This is known as the counterfactual.Establishing the counterfactual is not easy, since by definition it cannot be observed.A strong evaluation is successful in isolating the effect of the Resource from all other potential influences, thereby producing a good estimate of the counterfactual.
When communicating the impact of ELIXIR's resources and their role in accelerating science to funders and the public, the indicators should be relevant to the audience.This can be done by presenting them within a context that is readily understandable.

Detailed description of the indicators and related methodology
One of the challenges of data-intensive science is to facilitate knowledge discovery by assisting humans and machines in their discovery of, and access to, scientific data.FAIR is a set of guiding principles to make data Findable, Accessible, Interoperable, and Reusable 6 .These indicators will be used to demonstrate that ELIXIR Core Data Resources are compatible with the FAIR data principles.The A set of key indicators may be used to make a case for a Core Data Resource.Indicators aim to reflect the essence of the definition of an ELIXIR Core Data Resource and support the promotion of excellence in resource development and operation.
Indicators are grouped in five categories: (1) Scientific focus and quality of science The indicators recognise the heterogeneous nature of biological data, and the diversity of the supporting data resources, use cases, and communities served.Indicators can be used to measure technical and/or scientific readiness of a resource compared to defined quality standards.
The Table below maps indicators to corresponding FAIR criteria.
As the context of a core resource is critical to understanding its importance, indicators alone are not sufficient.Qualitative evidence is needed so that the resource can be reviewed throughout its life-cycle through the expert judgment of the ELIXIR Heads of Nodes and Scientific Advisory Boards.

Indicators and Related Information.
All elements in sections 1-4 require a response.
Quantitative indicators are underlined.

Scientific focus and quality
a. Archives vs knowledge bases: is the resource archival (taking submissions) or a knowledge base (added-value)?b.Scope statement: describe the scientific coverage and comprehensiveness of the resource.For example, all species or a subset of species, families, outputs from a particular experimental method?What is position of the resource with respect to other similar data resources?c.International dimension: does the resource have a global footprint?(Demonstrated through, for example, an international consortium delivering the resource, geographical diversity in sources of submissions, global literature curated, international diversity of delivery partners and/or funders) c.Privacy policy: does the resource have a publically available privacy policy in which security around personal data and cookies are described?d.Ethics policy: does the resource have an ethics policy that complies with all relevant international standards and best practices?
e. Sustainable support and funding: demonstrate the past and future funding and/or other commitments that support the resource by the host institution and/or other entities.

Impact and translational stories a.
Counterfactual: what would be the impact on the scientific community if the resource had not existed, or were to disappear and not be replaced?Is the resource globally unique?What would the impact on other dependent resources be?
b. Accelerating science: how does the resource accelerate science?For example, does the resource set standards; promote reuse of data or software; promote research efficiencies; extend technical products in other areas?c.Translational data: are there 'translational' figures familiar to the audience that will help them grasp the core nature of the resource?

Definition of terms used to measure overall resource usage (see 2.a)
Visits: a visit, or session, is a set of requests/interactions by a uniquely identified client within a specific time (typically, 30 minutes).The number of visits/ sessions is a measure of website traffic.
Unique Visitors: the number of visitors (unique IP addresses, unique visitors, or visitors) measures how many individuals access a website in a specified time, regardless of how often they visit.It can be determined in different ways.For example, number of: unique IP addresses, user cookies, unique IP addresses + user agent (a 'user agent' is the client that is used to access a web site.
Hits: can be used to analyse usage trends of a web resource.Hits measure the number of files downloaded when a web page is viewed.A web page is typically made up of a number of individual files, such as HTML documents, images, JavaScript files.When a web page is viewed, each file is requested from the server, adding to the hit-count.
Page views: (or pages, impressions or URLs): a request to load a single HTML file (web page) of a web site, identified by the URL in a browser.During a single visit, several different pages may be accessed.
Downloads: measures the data downloaded from a resource in volume/bandwidth, often in GigabytesGB.

Seed list of ELIXIR Core Data Resources
Using the definition of ELIXIR Core Data Resources above, we identified a 'seed list' of candidate core resources (Table 2) to inform Core Data Resource indicators.
Identifying ELIXIR Core Data Resources Identification of ELIXIR Core Data Resources involves a careful evaluation of the multiple facets of the data resources.This paper describes the overall approach for the selection of Core Data Resources, which will evolve over the coming months as the principles described in this paper are put into practice for the first time.
Indicators used are described in Box 1.The relevant ELIXIR Node submits the completed 'Case Document' (Box 2) to the ELIXIR Hub.
Cameron 5 ), the indicators will be used to inform a peer-review process described below.
A carefully chosen set of qualitative and quantitative indicators, tailored to bioinformatics resources, will inform identification of the ELIXIR Core Data Resources.The indicators will support, but not supplant, expert judgment.The ELIXIR Scientific Advisory Board will review the process in March 2017.In this initial test process and evaluation, up to 20 ELIXIR Core Data Resources will be selected.
Based on the experience from this first selection round, the Heads of Nodes Committee may recommend a refinement of the indicators and the overall process for future.Should the process prove to be sufficiently robust, the selected candidate resources will become the first set of ELIXIR Core Data Resources.The ELIXIR Scientific Advisory Board also reviews the portfolio of ELIXIR Core Data Resources and provides ongoing advice on the process for their identification.
As each ELIXIR Core Data Resource already has a governance structure that includes an independent, international Board, this individual review is not duplicated by the ELIXIR Advisory Board.The outcome is presented to the ELIXIR (governance) Board for information and to ensure that the process has been correctly applied.
Through the work of the Nodes, Advisory Board and the ELIXIR Hub, standardized data on indicators can also be collected and monitored.
In collaboration with the Nodes, monitoring data will be automatically collected at the ELIXIR Hub on an ongoing basis and will be regularly transmitted to the Heads of Nodes.Nodes undertake to provide the necessary data to the specification defined.
Reviewing the ELIXIR Core Data Resources ELIXIR Core Data Resources may be requested to report regularly on certain indicators, and to provide updates on any major changes.
The Heads of Nodes meeting will review all ELIXIR Core Data Resources every two to three years.However, a minimum of three Heads of Nodes may request an extraordinary evaluation of an individual resource, in particular, on the basis of the monitoring data.If the review raises issues concerning an ELIXIR Core Data Resource, the Heads of Nodes Committee is responsible for determining what action should be taken.

Discussion
Supporting the sustainability of data resources and how defining identifying them will contribute to science policy ELIXIR Core Data Resources form the centre of ELIXIR's sustainability strategy.The collected key indicators for these bioinformatics resources, and more specifically the impact and translational stories, will be used to make a case to funders.This information will in turn help them to translate the impact that Core Data Resources make.
Impact evaluation attempts to provide a definitive answer to the question of whether the resource is meeting its objective of fulfilling a specific need of the scientific community.The translational stories relate to the role of the resource in accelerating science and are thus a very important indicator.
In addition, the ELIXIR Core Data Resources could contribute to impact and econometric analysis of life-science data within ELIXIR, as well as events focused on communicating the value of sustainable infrastructure for open data to the European Commission and other stakeholders.

Capacity building
Core Data Resources will act as flagships of excellence.The use of defined indicators, in particular those around user policies and procedures, will be useful as benchmarks of quality and will support capacity building within the ELIXIR Community.
For example, the ELIXIR Core Data Resources, especially the knowledge bases, can function as 'concept authorities' within and beyond ELIXIR, having a clear role in standardising what the community understands by a given biological concept.
Certain additional indicators could be used outside of ELIXIR (e.g.uptime) to consolidate confidence across a wide range of stakeholders.This would require full transparency on how indicators are produced, so as to avoid misunderstanding or misuse.

Life-cycle management
Key indicators will inform life-cycle management, identifying trends and supporting decision-making around a given resource.

Conclusion
ELIXIR Core Data Resources form the centre of ELIXIR's sustainability strategy and science policy actions.The collected key indicators reflect the diversity of these bioinformatics resources, and will be used to make a case to funders.This information in turn will help them to translate the impact that Core Data Resources make.
Key indicators for Core Data Resources, in particular those around user policies and procedures, will be useful as flagships of excellence and best practice to support capacity building within the ELIXIR Community.The process may be extended to incorporate best practices on interoperability, on concept naming, identifier resolution, identifier mappings and data identity provision and protection.
The key indicators will inform life-cycle management, identifying trends and supporting decision-making around a given resource.This is important not only for the teams managing the resources, but also for the identification of Emerging Services that may evolve into Core Data Resources.As new resources are listed on the ELIXIR Node Service Delivery Plans, the indicators and capacity building around the Core Data Resources will support the growth of Emerging Services as they mature.
As ELIXIR continues to mature, the framework for life-cycle management will be put into practice, supporting the Emerging Services, and strengthening the ELIXIR infrastructure by creating a stairway to excellence.
The use of both quantitative and qualitative indicators reflects the need to understand the context in which resources operate, providing a clear and rational basis for efforts to strengthen resources and improve capacity building.Establishing the portfolio of ELIXIR Core Data Resources and ELIXIR Services is a key priority for ELIXIR and publicly marks the transition towards a cohesive infrastructure.b.Scope statement: describe the scientific coverage and comprehensiveness of the resource.For example, all species or a subset of species, families, outputs from a particular experimental method?How is the resource positioned with respect to other similar data resources?c.International dimension: does the resource have a global footprint?(e.g.demonstrated through an international consortium delivering the resource, geographical diversity in the source of the submissions, global literature curated, international diversity of delivery partners and/or funders) c.Privacy policy: does the resource have a publically available privacy policy in which security around personal data and cookies are described?
d. Ethics policy: does the resource have an ethics policy that complies with all relevant international standards and best practices?
e. Sustainable support and funding: demonstrate the past and future funding commitments and/or other commitments that support the resource by the host institution and/or other entities.

Impact and translational stories
a. Counterfactual: what would the impact on the scientific community be if the resource had not existed or was to disappear and not be replaced?Is the resource globally unique?What would the impact on other dependent resources be?
b. Accelerating science: how does the resource accelerate science?For example, does the resource set standards; promote reuse of data or software; promote research efficiencies; extend technical products in other areas?
c. Translational data: are there 'translational' figures that are familiar to the audience that will help them grasp the core nature of the resource? 1. 2.

Open Peer Review
Current Referee Status: If one is very familiar with Elixir, it probably makes sense, but if one isn't, then it is a bit confusing to read.
But it contains valuable information that I think would be generally useful to everyone trying to develop methods for evaluating data resources.In fact, an RFI put out by the US National Institutes of Health on repository metrics just closed today.So if the authors are willing, I think that providing some modifications would make the report more readable to a general audience.It would be nice if the first paragraph introduced Elixir a bit more and explained its structure.This could be done through either a diagram or a reference.But the Hub idea is critical to the governance of Core Data Resources proposed and it would be nice to make the structure clear.
The tense of the article is a bit unclear.Are there already approved Core Data Resources that have been evaluated by the criteria outlined?At times, it seems that way and other times, it seems like the process has not yet been implemented.In Table 2, some examples that are considered "core" are given.But in the text, it says that the resources in table 2 were identified as a "seed list" to inform Core Data Resources.So it implies that they haven't yet gone through the process.I think stating up front where you are in the process would make it less confusing.
Methods section: The term "indicator" is first used in second paragraph of the first section of methods section, but is not defined until the next section.It should be defined earlier.
"Legacy stage" is used in the 3rd paragraph.Legacy has a meaning in data-already existing-and so I think some definition is required here.It is, in fact, defined in Table 1, so a reference to the Table would be sufficient.
The indicators are listed in the main text, again in Box 1 and again in Fig 2 .Seems like a bit of overkill.Also, the purpose of the indicators is defined in multiple places and it is a bit repetitive, e.g., the explanation given under the section "Detailed description of the indicators and related methodology" really isn't necessary.
FAIR is introduced in Box 1, but not in the text at all.It seems like it should be mentioned in the

7.
FAIR is introduced in Box 1, but not in the text at all.It seems like it should be mentioned in the introduction to indicators in the methods as it is used as a set of criteria throughout Box 1.
Is there a timeline as to when the first core resources will be approved (if they haven't been already-see pt 2) I have read this submission.I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
No competing interests were disclosed.Competing Interests:

Figure 1 .
Figure 1.The place of ELIXIR Services and ELIXIR Core Data Resources within ELIXIR's mission.

( 2 ) 3 ) 5 )
Community served by the resource (Quality of service (4) Legal and funding infrastructure, and governance (Impact and translational stories.

ELIXIR
Core Data Resources should each have an international independent Scientific Advisory Board.Such boards are made up of distinguished academic and industry researchers and professionals who conduct scientific and/or technological review, ensuring quality and providing strategic advice to resource managers.Identification of ELIXIR Core Data Resources does not encroach on these governance structures.The establishment of Scientific Advisory Boards for Core Resources and Nodes is among the best practices that will be promoted by the Node Capacity Building and Communities of Practice.Indicators can only be useful if they are underpinned by an open, transparent and coherent collection infrastructure, so clear methods of collection and processing are needed.

Figure 2 .
Figure 2. A carefully chosen basket of qualitative and quantitative indicators for bioinformatics resources.

Figure 3 .
Figure 3. Process for identifying ELIXIR Core Data Resources.

Box 2 .
Case Document TemplateA 'Case Document' describes a (candidate) Core Data Resource and is based on the indicators introduced in Box 1.
Department of Neurosciences, Center for Research in Biological Systems, University of California, San Diego, La Jolla, CA, USA Hypothes.is,San Francisco, CA, USA Founder, SciCrunch.com,San Diego, CA, USA I understand that this article is included in the Elixir report collection and it does, in fact, read like a report.

) Quality of service Certain
service levels and reliability can be quantified with specific technical indicators such as: the uptime of the resource; response times; availability and periodic application of meaningful and automated tests; user support and related training; use of community-recognised standards; diversity of data retrieval mechanisms; and other services.Usually, this requires a quality-assurance process during service development and operation.The Accelerating the ELIXIR Training Programme and the ELIXIR Training Platform will support resources delivering training, as well as provide good-practice guidelines and systems for evaluation.(4)Legal and funding infrastructure, and governance As stable research infrastructures, Core Data Resources can demonstrate that they have a sound legal, funding and governance structure.

Table in
Box 2 presents a 'Case Document' template for describing a data resource using these indicators.Indicators to support expert judgmentTaking into account that 'Not everything that can be counted counts, and not everything that counts can be counted' (William Bruce Box 1.

Quantitative and qualitative indicators for ELIXIR Core Data Resources ELIXIR
Core Data Resources are defined as a set of European data resources that are of fundamental importance to the broad life-science community and the long-term preservation of biological data.
1verall usage: what is the usage of the resource for the past 2-3 years?i.Access via a web browser: number of visits, unique visitors, hits, and page views1ii.Access via additional access methods: visits, unique visitors, hits, and downloads (includes FTP downloads and programmatic access) b.Potential usage: what is the estimated size of the global potential user community?c.Usage in research as measured through citation in the literature: i. Citation of a resource name: the number of times the resource name is mentioned in scientific articles per year (in Europe PMC) ii.Citation of data of a resource: the number of times accession numbers from the resource are mentioned or cited in research articles (in Europe PMC) Identifier use: does the resource provide persistent and unique identifiers?b.Data throughput: number of entries, depositions (records or bytes ingested per year), records processed, genomes assembled, etc. per year, for past 2-3 years.Use of standards: which community-recognised standards are used for metadata and data (e.g.MIAME, JATS, INSDC features, ontologies)?Provide a link to documentation.e. Links to documentation of provenance: does the resource link to the scientific literature for provenance of facts or biological context?
d. Staff effort: number of FTEs per year for the past 2-3 years iii.Key publications describing the resource list (e.g.publications in NAR Database issue) and the number of citations (in Europe PMC).d.Dependency of other resources: do other resources have a dependency on the resource described here to provide that service (i.e.what is the reach-through)?3. Quality of service a. i.Data sharing services: list services through which data is shared (e.g.website, APIs, FTP, TripleStore) ii.Data sharing formats: list formats for available data (e.g.plain text, FASTA, XML, RDF, Dublin Core, tsv, JSON) g.Customer service i.Helpdesk: does the resource run a helpdesk?ii.User feedback: does the resource seek and incorporate user input into service design decisions?iii.Training: does the resource undertake training?4. Legal and funding infrastructure, and governance a. Scientific Advisory Board: does the resource have an international, independent Scientific Advisory Board b.Open Science: does the resource have a legal framework that supports Open Science? e.g.open licenses or a public statement of open terms of use.

Table 2 . Examples of European data resources that are considered as core for the life-science community. Resource name Institutions Type
Only data resources that are part of an ELIXIR Node Application and/or Service Delivery Plan (in the case of EMBL-EBI, the 'Work Programme') can be candidate ELIXIR Core Data Resources.Service Delivery Plan (or Work Programme).The ELIXIR Hub has an advisory role in selecting ELIXIR Core Data Resources.The Hub has no decision-making power and does not evaluate the proposals.The ELIXIR Director informs the Heads of Nodes Committee of the candidate ELIXIR Core Data Resources.The Heads of Nodes Committee can request additional information about a candidate Resource from the relevant Head of Node.The Heads of Nodes Committee convenes annually in person to review submitted Case Documents and determine the list of ELIXIR Core Data Resources.The initial selection is expected to grow with time.
UniProt EMBL-EBI (European Molecular Biology Laboratory -European Bioinformatics Institute); SIB Swiss Institute of Bioinformatics; Protein Information Resource (PIR) -Georgetown University Medical Centre This is important not only for the resource teams, but also for identifying Emerging Services that may evolve into ELIXIR Services.As new resources are listed in the ELIXIR Node Service Delivery Plans, indicators and capacity building around the Core Data Resources will support Emerging Services as they mature.ELIXIR will strive to add value to all ELIXIR resources, including ELIXIR Services, by supporting interactions of the Core Data Resources with one another and with ELIXIR Services and Emerging Services for the benefit of the larger user community.Examples of this are use-case driven enhancement of the interoperability of the ELIXIR Core Data Resources with one another and with other ELIXIR supporting helpdesks to scale national operations, and implementation studies to explore links to national infrastructures and data services.

Legal and funding infrastructure, governance a. Scientific Advisory Board: does
ii. Response times of key web pages.d.Use of standards: which community-recognised standards are used for metadata and data (e.g.MIAME, JATS, INSDC features, ontologies)?Provide a link to documentation.e. Links to documentation of provenance: does the resource link to the scientific literature for provenance of facts or biological context?f.Data availability -access services and formats: i.Data sharing services: list services through which data is shared (e.g.website, APIs, FTP, TripleStore) ii.Data sharing formats: list formats data is available in (e.g.text, FASTA, XML, Dublin Core, tsv, JSON) g.Customer service: i. Helpdesk: does the resource operate a helpdesk?ii.User feedback: does the resource seek and incorporate user input into service design decisions?iii.Training: does the resource undertake training activities? 4. the resource have an international, independent Scientific Advisory?b.Open Science: does the resource have a legal framework that supports Open Science? e.g.open licenses or public statement of open terms of use.