ALL Metrics
-
Views
-
Downloads
Get PDF
Get XML
Cite
Export
Track
Method Article

An open and transparent process to select ELIXIR Node Services as implemented by ELIXIR-UK

[version 1; peer review: 1 approved, 2 approved with reservations]
PUBLISHED 21 Dec 2016
Author details Author details
OPEN PEER REVIEW
REVIEWER STATUS

This article is included in the Bioinformatics gateway.

This article is included in the ELIXIR gateway.

Abstract

ELIXIR is the European infrastructure established specifically for the sharing and sustainability of life science data. To provide up-to-date resources and services, ELIXIR needs to undergo a continuous process of refreshing the services provided by its national Nodes. Here we present the approach taken by ELIXIR-UK to address the advice by the ELIXIR Scientific Advisory Board that Nodes need to develop “mechanisms to ensure that each Node continues to be representative of the Bioinformatics efforts within the country”. ELIXIR-UK put in place an open and transparent process to identify potential ELIXIR resources within the UK during late 2015 and early to mid-2016. Areas of strategic strength were identified and Expressions of Interest in these priority areas were requested from the UK community. A set of criteria were established, in discussion with the ELIXIR Hub, and prospective ELIXIR-UK resources were assessed by an independent committee set up by the Node for this purpose. Of 19 resources considered, 14 were judged to be immediately ready to be included in the UK ELIXIR Node’s portfolio. A further five were placed on the Node’s roadmap for future consideration for inclusion. ELIXIR-UK expects to repeat this process regularly to ensure its portfolio continues to reflect its community’s strengths.

Keywords

ELIXIR, ELIXIR-UK, e-Infrastructure, ESFRI

Introduction

ELIXIR, the European infrastructure for life science data1, is made up of individual Nodes, one for each of the organisation’s constituent members (20 as of 1st November 2016: Belgium, Czech Republic, Denmark, EMBL-EBI, Estonia, Finland, France, Germany, Ireland, Israel, Italy, Luxembourg, Netherlands, Norway, Portugal, Slovenia, Spain, Sweden, Switzerland and the UK), and a coordinating hub. The individual ELIXIR Nodes provide the services and resources that support the five pillars of ELIXIR (Compute, Tools, Data, Interoperability and Training infrastructures).

ELIXIR nodes need to be able to evolve their contributions to ELIXIR by bringing new services and resources. ELIXIR identifies two types of service: Node-funded services, which are funded nationally and are contributed to ELIXIR from a national Node; and Commissioned Services, which are funded by ELIXIR as a whole via the ELIXIR Hub. In some ELIXIR Nodes, Node-funded services receive funds through their national Nodes; in the case of the UK’s Node, ELIXIR-UK, resource funding is through direct grant funding to resources and services from the national funders. In ELIXIR terms, these are still labelled as “Node-funded”. The process described in the present article was set up to identify Node-funded services and resources for ELIXIR-UK. ELIXIR sets high standards for the services it provides. Consequently, nodes need to take full account of these requirements when selecting and proposing their services, which are ultimately judged for suitability by the ELIXIR Scientific Advisory Board (SAB) and Board of ELIXIR (see the online ELIXIR Handbook for more detail).

ELIXIR-UK was established in September 2013, and as its first contribution to ELIXIR took on a thematic focus, namely of coordinating training activity. More recently, it has sought to expand its remit. To address the SAB’s recommendation that Nodes put in place “mechanisms to ensure that each Node continues to be representative of the Bioinformatics efforts within the country”, ELIXIR-UK developed a process to choose new services and resources to add to its existing portfolio. Its aims in developing this process were to:

  • Reflect national strengths and priorities in bioinformatics

  • Engage its national community

  • Build a robust, transparent and open process that its community would regard as fair and could continue to be applied to allow the Node to develop over time.

Process overview

As illustrated in Figure 1, the process implemented by ELIXIR-UK went through seven key phases, which are expanded on in the following sections:

  • 1. Strategic prioritization

  • 2. Identifying possible candidate resources

  • 3. Setting up appropriate structures

  • 4. Establishing assessment criteria

  • 5. Engaging the community

  • 6. Assessing Expressions of Interest

  • 7. Finalising a new portfolio

1eedaa73-cb6b-4d56-8e63-f506da41a098_figure1.gif

Figure 1. Summary view of the information flow in the ELIXIR-UK Node Expansion process.

Items in the red correspond to the numbered list of key phases. Numbers in red correspond to the seven phases of the process listed under the section “Process overview”.

Strategic prioritization

The requirement to ensure that each Node continues to be representative of the Bioinformatics efforts within the country could be seen as open ended, and thus could ultimately lead to an ill-focussed collection of resources and services. To avoid this, ELIXIR-UK identified a set of priority areas within which to focus submissions to the process. These were initially identified by discussions within the Node and were refined by discussion with the Node’s funding organisations (Biotechnology and Biological Sciences Research Council [BBSRC], Medical Research Council [MRC] and Natural Environment Research Council [NERC]) and with the Scientific Development Group (SDG), which was a community body set up by the Node that is tasked with identifying new node resources (see below).

As a consequence of these discussions, Expressions of Interest (EoIs) were invited in the following priority areas, identified as being of high strategic importance within the UK:

  • Human clinical and health ‘omics and related areas in health informatics

  • Agricultural ‘omics and related data resources

  • Image informatics (including atlases)

  • Structural bioinformatics

  • Technical infrastructure for interoperability and training including standards

Identifying possible candidates

ELIXIR-UK aimed to reconcile two potentially conflicting drivers in developing its expansion process. Firstly, it wanted to be as open to the UK bioinformatics community as it could. This is an ongoing challenge because a) ELIXIR has incomplete brand recognition within the UK community, and b) is not well-regarded by some, being seen either as a closed club or unproductive. Secondly, the Node wanted to ensure it received Expressions of Interest from potential services and resources that were demonstrably of high value to the international life sciences community. To address these requirements, the Node approached the recruitment of potential candidates in two ways. Firstly, it publicised its “Node Expansion” process well in advance using its web site, Twitter and word-of-mouth. Secondly, it sent targeted emails to potential candidate resources. These were identified using a variety of inputs:

  • Brainstorming by members of the existing node

  • Setting up a specific working group - on Agriculture-related data - for an area that was not well-represented in the current node

  • Additional suggestions from the SDG (see next section) and funders

The Scientific Development Group

The key body in the Node’s expansion process was the SDG. This was set up by the Node to evaluate EoIs to join the Node against a set of published criteria (see below). This group was also involved in refining those criteria and providing suggestions of resources to be invited to provide EoIs.

The membership of this group was based on suggestions from within the Node and from its funders. The group’s composition reflected the priority areas identified for the expansion, geographic spread, and the inclusion of at least one industry and at least one overseas representative. The Chair was chosen for his experience as a senior officer of a UK funding agency and knowledge of appropriate processes for activities of this kind. For the record we note that the group did not have an appropriate gender balance (it was 100% male). This is a defect we intend to remedy in future.

Assessment criteria

Over time ELIXIR has been evolving both its classification of resources and its criteria for selecting them. During the period of the UK Node’s expansion process these definitions and criteria continued to evolve. The assessment criteria developed by ELIXIR-UK were developed by internal discussion and in discussion with the SDG and were also discussed informally with the leaders of Work Package 3 of the EXCELERATE programme (Jo McEntyre and Christine Durinx), as their criteria developed in parallel. The final set of criteria, which were provided to applicants as an openly shared Google document, were:

  • Alignment with the five ELIXIR infrastructure themes (data, tools, compute, interoperability, training)

  • Strong complementarity to the 2014-18 ELIXIR programme

  • Complementarity to ELIXIR-UK strategic themes

  • Potential for cross-Node collaborations

  • Provision of comparable impact to existing ELIXIR resources from other Nodes already accepted by the ELIXIR SAB

  • Resource contribution to wider EU infrastructures and integration

  • Ability to interoperate with other ELIXIR resources

  • Evidence of community outreach and adoption

  • Leadership in data stewardship within a community

  • Evidence of long-term sustainability

To facilitate applicants demonstrating that their resources fulfilled these criteria, an Expression of Interest template form was provided, also via Google documents.

The criteria developed by EXCELERATE Work Package 3 have subsequently been finalised and form the basis of the ELIXIR process for selection of Core Data Resources2.

Engaging the community

As outlined, it was important to ensure community buy-in to this process (in order to ensure that the Node was able to engage sufficient high quality resources) and at the same time it was important to be sure that community members who might be interested in participating in ELIXIR-UK were aware of what was required and the expectations that would be placed on them as ELIXIR-UK Node resources. Formal community engagement took place in two phases: a webinar, led by the Head of Node (CAG) and Node Coordinator (JMH), in February 2016 and a workshop, hosted at the Wellcome Trust building in London, in March 2016. The aim of the webinar was to introduce ELIXIR and ELIXIR-UK and the rationale behind the node expansion process. The aim of the workshop was to introduce and discuss the assessment criteria in detail, so that potential applicants could be clear as to what was required. The presentation given at the workshop is available via Slideshare. At this stage a deadline was set for the receipt of Expressions of Interest by the Node. It is worth noting that the deadlines for the process were tight: EoIs were requested by the end of March 2016 and the assessment meeting took place at the end of April with some iterations taking place in May. We were fortunate in being able to run such a tight schedule due to a) clear and lightweight requirements for the EoIs; b) what we believe to be clear and effective communications; and c) motivated applicants and SDG members.

Assessment of Expressions of Interest

EoIs were assessed by the SDG against the published set of criteria. To facilitate assessment of EoIs, three group members were allocated to each EoI (18 were submitted). The three members were asked to score EoIs from 1 to 4: 1 = ready for inclusion in ELIXIR (“infrastructure ready”); 2 = further discussion or clarification needed; 3 = not ready, but suitable enough to be placed on a roadmap for future inclusion; 4 = not suitable. The assessments for each EoI were introduced by one member of the group leading on to an open discussion. Representatives of the Node funders and the ELIXIR-UK executive observed the meeting to give advice on strategic alignment. EoIs were given a consensus final score using the same scale as before, with a score of 2 in this case representing the need for further clarification of issues raised by the group. Resources given a 2 rating were asked for further information, which led to their final score being revised upwards or downwards in a subsequent iteration.

Results of the assessment

The outcomes of the assessment are summarised in Table 1. Nine EoIs were considered to be infrastructure ready (score of 1) on the first pass of assessment, and a further five were asked for more detail on their proposal (score of 2).

Table 1. Summary of outcome of the Scientific Development Group deliberations.

The table gives numbers of proposals classified as 1 (ready for inclusion in ELIXIR (“infrastructure ready”); 2 (further discussion or clarification needed); 3 (not ready, but suitable enough to be placed on a roadmap for future inclusion); 4 (not suitable).

RatingAfter panel
assessment
After
iteration
11013
251*
345
400

*In this case the group were unclear whether the proposed resource could be included in the Node’s offering. This case was put forward to the ELIXIR SAB for further input who recommended it be placed in Category 1.

An iteration of discussions with resource scientists allowed questions raised by the SDG to be considered further. Where these were answered satisfactorily, resources were moved up to infrastructure-ready status. Otherwise they were put on the roadmap or, in one case, referred to the ELIXIR SAB for further comment (in this latter case, SAB guidance subsequently resulted in it being accepted as infrastructure ready).

After ratification by the ELIXIR-UK executive and notification to the ELIXIR Hub, highly rated resources were included directly into the Node’s portfolio and were included in the Node Application presented to the ELIXIR SAB in June 2016 and the ELIXIR Board in November 2016. Others were placed on the Node’s roadmap for possible future inclusion.

The services and resources selected as ready for immediate inclusion are listed in Table 2.

Table 2. Resources judged to be ready for inclusion in ELIXIR at the end of the assessment process.

Resources are classified by strategic themes within ELIXIR-UK.

NameStrategic themeURL
Ensembl – farmed and
domesticated animals3
Agri-food datawww.ensembl.org
Pathogen Host
Interactions Database
(PHI-base)4
Agri-food datahttp://www.phi-base.org
Biomedical Atlas CentreGene expression
atlases
http://emouseatlas.org/,
http://hudsen.org/,
http://echickatlas.org/
IUPHAR/BPS Guide to
Pharmacology5
Human health
& disease data
www.guidetopharmacology.org
BioSharing6Interoperability
services
https://biosharing.org
InterMine7Interoperability
services
http://www.intermine.org,
https://github.com/intermine/intermine
ISA Tools & Commons8Interoperability
services
http://www.isa-tools.org
http://www.isacommons.org
CATH-Gene3D9Protein structure
& function
http://www.cathdb.info/
Jalview and the Dundee
Resource for Sequence
Analysis and Structure
Prediction10
Protein structure
& function
http://www.jalview.org
www.compbio.dundee.ac.uk/jpred
Phyre211Protein structure
& function
www.imperial.ac.uk/phyre2
Birmingham Metabolomics
Training Centre
Traininghttp://www.birmingham.ac.uk/facilities/metabolomics-training-
centre/index.aspx
Cambridge Bioinformatics
Training Programme
Traininghttp://bioinfotraining.bio.cam.ac.uk/
http://training.csx.cam.ac.uk/bioinformatics/Event-timetable
Edinburgh Genomics
Advanced Training in
Bioinformatics
Traininghttps://genomics.ed.ac.uk/services/training
TeSS (Training e-Support
System)
Traininghttps://tess.elixir-uk.org/

*References for the databases/tools have been added where available.

Future activities and conclusions

We believe that the process outlined here was open, transparent and fair. We note that the “success rate” of the process was high. No resources were rejected outright and more than 70% were promoted immediately to the Node’s portfolio. This does not reflect a lax process, but is likely to have had a number of contributing factors, including:

  • The fact that this was the first call of this kind meant that the Node could call on a number of outstanding, internationally-acknowledged resources. The resources placed on the roadmap were generally also well regarded, but usually in an early phase of their development. Our expectation is that most of these will be recognised as Node-funded resources in future.

  • There was a clear explanation and open presentation of the high standards expected of successful resource. Therefore, it is likely that only resources that considered they had a realistic chance of success after the webinar and workshop put their names forward. Consequently, we did not receive any truly speculative proposals.

Another aspect of the process we outline here is the short time period over which it was carried out. In particular, resources were only given four weeks to submit EoIs. A number of features of the process facilitated this: clear timelines, clear guidance as to what was required, the availability of a template for EoIs that helped proposers to compile their EoIs, and lightweight requirements for completing EoIs, which were nevertheless sufficient to allow the SDG to carry out its work effectively. Engagement at a senior level by both the Node and proposers was also important. It was also important to organise meetings, especially of the SDG, sufficiently ahead of time to allow members to both assess the EoIs and attend the meetings, either in person or remotely.

To maintain and continue to improve the Node’s alignment with UK research strengths, it plans to hold regular refresh exercises to introduce new resources into the Node. Plans for how this will be done are currently under development. To pursue this process we expect that we will need to develop community engagement in the specific priority areas, so that potential proposers are primed.

Comments on this article Comments (0)

Version 2
VERSION 2 PUBLISHED 21 Dec 2016
Comment
Author details Author details
Competing interests
Grant information
Copyright
Download
 
Export To
metrics
Views Downloads
F1000Research - -
PubMed Central
Data from PMC are received and updated monthly.
- -
Citations
CITE
how to cite this article
Hancock JM, Game A, Ponting CP and Goble CA. An open and transparent process to select ELIXIR Node Services as implemented by ELIXIR-UK [version 1; peer review: 1 approved, 2 approved with reservations]. F1000Research 2016, 5(ELIXIR):2894 (https://doi.org/10.12688/f1000research.10473.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?
Key to Reviewer Statuses VIEW
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Version 1
VERSION 1
PUBLISHED 21 Dec 2016
Views
44
Cite
Reviewer Report 23 Jan 2017
Janet Kelso, Department of Evolutionary Genetics, Max Planck Institute for Evolutionary Anthropology, Leipzig, Germany 
Approved with Reservations
VIEWS 44
This manuscript describes the process whereby the UK node of ELIXIR identifies and prioritises services for inclusion in the node.

It is an important contribution as it is the first time a strategy for service identification and ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Kelso J. Reviewer Report For: An open and transparent process to select ELIXIR Node Services as implemented by ELIXIR-UK [version 1; peer review: 1 approved, 2 approved with reservations]. F1000Research 2016, 5(ELIXIR):2894 (https://doi.org/10.5256/f1000research.11283.r18696)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 27 Mar 2017
    John Hancock, ELIXIR, UK
    27 Mar 2017
    Author Response
    In response to this review we have:
    • Added a section on the extent to which this approach is generalizable to other nodes, and on whether there are aspects that are
    ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 27 Mar 2017
    John Hancock, ELIXIR, UK
    27 Mar 2017
    Author Response
    In response to this review we have:
    • Added a section on the extent to which this approach is generalizable to other nodes, and on whether there are aspects that are
    ... Continue reading
Views
47
Cite
Reviewer Report 09 Jan 2017
Alfonso Valencia, Life Science Department, Barcelona Supercomputing Centre - Centro Nacional de Supercomputación, Barcelona, Spain 
Approved with Reservations
VIEWS 47
The report describes the process of selection of the UK-ELIXIR xxxx to be presented to the ELIXIR SAB.
The report is very informative and being the first node that describes in detail their national process, it has the potential to ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Valencia A. Reviewer Report For: An open and transparent process to select ELIXIR Node Services as implemented by ELIXIR-UK [version 1; peer review: 1 approved, 2 approved with reservations]. F1000Research 2016, 5(ELIXIR):2894 (https://doi.org/10.5256/f1000research.11283.r18698)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 27 Mar 2017
    John Hancock, ELIXIR, UK
    27 Mar 2017
    Author Response
    In response to this review:
    • We note that the composition of the Scientific Development Group is listed in the Acknowledgements
       
    • Revised Figure 1 to put
    ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 27 Mar 2017
    John Hancock, ELIXIR, UK
    27 Mar 2017
    Author Response
    In response to this review:
    • We note that the composition of the Scientific Development Group is listed in the Acknowledgements
       
    • Revised Figure 1 to put
    ... Continue reading
Views
44
Cite
Reviewer Report 30 Dec 2016
Christine Durinx, SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland 
Approved
VIEWS 44
The article is clear and well written. The figure and tables support the content.

Some suggestions, mainly to improve clarity:

Introduction

- It would be helpful to mention that the ELIXIR ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Durinx C. Reviewer Report For: An open and transparent process to select ELIXIR Node Services as implemented by ELIXIR-UK [version 1; peer review: 1 approved, 2 approved with reservations]. F1000Research 2016, 5(ELIXIR):2894 (https://doi.org/10.5256/f1000research.11283.r18697)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 27 Mar 2017
    John Hancock, ELIXIR, UK
    27 Mar 2017
    Author Response
    In response to this reviewer:
    • As also requested by another reviewer we now mention that ELIXIR Services can be data resources, tools, and services
       
    • Included
    ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 27 Mar 2017
    John Hancock, ELIXIR, UK
    27 Mar 2017
    Author Response
    In response to this reviewer:
    • As also requested by another reviewer we now mention that ELIXIR Services can be data resources, tools, and services
       
    • Included
    ... Continue reading

Comments on this article Comments (0)

Version 2
VERSION 2 PUBLISHED 21 Dec 2016
Comment
Alongside their report, reviewers assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Sign In
If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password.

The email address should be the one you originally registered with F1000.

Email address not valid, please try again

You registered with F1000 via Google, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Google account password, please click here.

You registered with F1000 via Facebook, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Facebook account password, please click here.

Code not correct, please try again
Email us for further assistance.
Server error, please try again.