Making European performance and impact assessment frameworks for research infrastructures glocal

Introduction

The long-term sustainability of Research Infrastructures^[1] (RIs) is of great importance to the European Strategy Forum on Research Infrastructures (ESFRI) and the European Union more broadly, as shown by calls for RIs to demonstrate their economic and wider benefit to society.¹ For the Organisation for Economic Co-operation and Development (OECD), sustainability is also a major concern as RIs represent an increasingly large share of research investment by national governments.² As a result, recent years have seen the emergence of a number of frameworks (ESFRI, OECD and that developed by the EU-funded RI-PATHS project) to guide RIs in their journeys to demonstrate performance and impact, going beyond simply scientific impact, and considering public value more generally.

Organization and historical path

Although BioData.pt is the Portuguese Infrastructure of Biological Data and the Portuguese Node of the ESFRI Landmark ELIXIR (a pan-European research infrastructure for life science data, Ref. 3), it relies on fixed-term project funds to operate. A historical note of BioData.pt and the Portuguese Node of ELIXIR is depicted in Figure 1.

Figure 1. Historical path of the Portuguese Node of ELIXIR.

In 2010, a Portuguese researcher from IGC (Instituto Gulbenkian de Ciência) at the European Bioinformatics Institute was challenged to participate in the onset of ELIXIR and create the Portuguese Node of this, yet to be, research infrastructure. In 2011, a first step was done, by creating the Portuguese bioinformatics woody plants platform followed by the creation of BioData.pt – Portuguese Biological Data Network, to operate the Portuguese Node of ELIXIR (ELIXIR PT), in 2013. In the same year, the ELIXIR, an intergovernmental organisation bringing together life science resources from across Europe, namely, databases, software tools, training materials, cloud storage and supercomputers, was founded. In 2014, the Fundação para a Ciência e a Tecnologia, on behalf of the Portuguese Government, signed the ELIXIR Consortium Agreement, by which Portugal adhered to ELIXIR and, in 2016, a collaboration agreement between INESC-ID (Instituto de Engenharia de Sistemas e Computadores, Investigação e Desenvolvimento) and ELIXIR was signed, creating the Portuguese Node of this pan-European infrastructure. The merger between BioData.pt and ELIXIR PT was formalized in 2018 and, very recently, already in 2021, the BioData.pt Association was founded as the legal entity of this RI.

Since 2017, the development and early operation of BioData.pt have been funded through a €2.7M project grant to last for four years, to lay the foundation of this RI. The “Building BioData.pt” project involved 11 beneficiaries and 81 participants, many of which providing in-kind expertise on computing, bioinformatics and data management to a range of end-users, as well as building and operating the infrastructure itself. Of the project budget, 33% was used to hire human resources and 57% to purchase equipment, mostly computing resources to assemble the nationally distributed computing infrastructure. The remainder was used in the adaptation of buildings to host the computing infrastructure or to deliver training. “Building BioData.pt” was structured as shown in Table 1.

Table 1. Working groups (WG) of the “Building BioData.pt” project, which was instrumental to the creation and early operation of the BioData.pt infrastructure.

Research Communities	Support to Research Communities	Industry Engagement
WG1 - Plants	WG6 - Common Infrastructure	WG8 - Industry & Entrepreneurship
WG2 - Marine Resources	WG7 - Training
WG3 - Systems biology	WG3 - Systems biology
WG4 - Neurosciences	WG10 - Project Management & Dissemination
WG5 - Yeastract

Methods

Based on the simplified RI-PATHS approach, which classifies indicator types in activity (concrete activities to be carried out within the scope of the project, visible to the public and under full control of the organization), outcome (short-term direct results of each activity, not under direct control of the organization) and impact (transformative effects of the activity on its target audience and beyond, in the mid to long term),⁴ a “BioData.pt Monitoring and Evaluation Matrix” (B-MEM) (Figure 2, Ref. 5) and supporting guidelines were developed to facilitate the onset of evaluation processes, by the BioData.pt infrastructure. This simple and participatory approach allowed the project management team (the project manager and the executive director) to view and document, for the first time, the 10 working groups (or work packages), involving about 50 researchers (Table 1) of “Building BioData.pt” in terms of activities, outcomes and impact, and align these with the overarching objectives of the project. In a second stage, the existing “Building BioData.pt” indicators, which had been maintained during the four years of grant execution (2017-2021), were categorized as activity, outcome and impact indicators, and assigned to the most relevant RI-PATHS impact area.

Figure 2. BioData.pt Monitoring and Evaluation Matrix.

This matrix, internally developed during the BioData.pt project, was used to understand and document project working groups in terms of activities, outcomes and impact.

Finally, each existing “Building BioData.pt” indicator, which could be quantitative or qualitative by nature, was documented and cross-checked against those compiled by the RI-PATHS project, and the OECD and ESFRI frameworks.

The following part of this study involved 3 additional experts (2 from BioData.pt and 1 from ELIXIR Hub) to reflect and perform a thorough analysis.

Taking advantage of the previous exercise, and to assess the commitment of BioData.pt with its purpose as a national RI and the Portuguese Node of ELIXIR, and its potential contribution to international initiatives such as EOSC and the UN Sustainable Development Goals (SDGs), the categorised “Building BioData.pt” indicators or proxies were assessed against several sets of strategic objectives relevant to research infrastructures and their work: those of BioData.pt (as an organisation), those of the current ELIXIR Scientific Programme,⁶ those of the Portuguese National Roadmap of Research Infrastructures from Fundação para a Ciência e a Tecnologia (FCT),⁷ those of EOSC⁸ and the SDGs.⁹ Arbitrary units (A.U.) were attributed to each of these dimensions, according to the level of BioData.pt commitments: objectives/goals of BioData.pt (4.5 A.U.), ELIXIR (2 A.U.), National Roadmap of RIs (2 A.U.), EOSC (1 A.U.) and the SDGs (0.5 A.U.), and a threshold was defined at 6.5 A.U., a score that could only be achieved if an indicator matched BioData.pt goals and, at least, those of ELIXIR or the National Roadmap of RIs, to which BioData.pt is bound by a formal relationship. Then, each indicator maintained during “Building BioData.pt” and categorised was analysed with respect to its contribution to the objectives of each of the former indicated organisations/initiatives.

To enable the visualisation of this analysis, a stacked column chart was generated entirely using the Rstudio software version 2021.09.0.351 and ggplot2 version 3.3.5,¹⁰ specifically using the ggplot() function, due to the complexity of the graphic. The fct_reorder() function of forcats version 0.5.1 was used to reorder indicators according to their relevance, in decreasing order.

Implementation and impact analysis

The first step was the description of the “Building BioData.pt” project working groups, in terms of activities, outcomes and impact, and their alignment with its two overarching objectives, using the BioData.pt Impact Assessment Matrix (B-MEM) and the respective guidelines. Working groups (WG) 1 to 5 were dedicated to domain specific activities which, overall, aimed to build the BioData.pt Research Communities, and their expected outcomes were data curation, integration and availability for domain-specific areas with impact in enhancing the quality of scientific research by promoting the FAIR principles. The global activity of WGs 6, 7, 9 and 10 was to set up the Support to Research Communities, having three outcome/impact lines: 1) a fully available computational infrastructure for data analysis/better and more extensive computing resources; 2) promotion of capacity-building in bioinformatics and research data management in the national research community/improved research efficiency & effectiveness and human capital; and 3) overall management of RI and operation of the Portuguese Node of ELIXIR/strengthened RI long-term sustainability. Both “Research Communities” and “Support to Research Communities” were developed under the objective “Strengthening research, technological development and innovation”. The second objective “Enhancing research knowledge transfer from academy to industry” was served by the activity Industry Engagement, developed by WG8, delivering promotion of knowledge transfer to industry to bring an added value to companies’ data and an industrial ecosystem of companies aware/beneficiary of bioinformatics and research data management as outcome and impact, respectively.

Yet with the help of the B-MEM, the “Building BioData.pt” project and Infrastructure reports were mined for indicators that would relate to those compiled by the ESFRI, OECD and RI-PATHS frameworks. Table 2 shows one example of this implementation, performed for the activity “Research Communities”, in relation to RI-PATHS. In detail, for this activity, we were able to find indicators of two types, activity and outcome, distinctly distributed by three of four impact areas, Human Resources, Economic and Innovation and Societal/Social. For each type of indicator identified, the indicator itself, the evidence for confirmation and method for description is listed. For instance, under the objective “Strengthening research, technological development and innovation”, in the activity “Research Communities”, the activity indicator Number of publications is identified by direct evidence (e.g., from Web of Science) and the method for collecting evidence is counting the number of publications. The full dataset of BioData.pt project indicators classified following the RI-PATHS approach can be accessed.¹¹

This exercise was repeated using the impact assessment frameworks of ESFRI and OECD. These approaches cover more explicitly the so-called scientific impact, of which some indicators, like Number of publications and Number of Citations, are also covered in the Human Resources area of the RI-PATHS approach.

A summary of RI-PATHS’ indicator types produced by “Building BioData.pt” (Table 3) shows that the different activities of the project, “Research Communities”, “Support to Research Communities” and “Industry and Entrepreneurship”, encompassing its 10 working groups had impact in all four RI-PATHS areas, namely, Human Resources, Economic and Innovation, Social/Societal and Policy, the working groups that provide “Support to Research Communities” having the broader socio-economic impact.

The indicators of “Building BioData.pt” identified in the ESFRI list corroborated the previous finding that the activities with broader impact were those dedicated to the “Support of Research Communities” (Table 4). The detailed information for this exercise is available for consultation.¹¹ In addition, “Building BioData.pt” generated indicators for all ESFRI’s areas and covered all indicators except number 14 (Number of publicly available data sets used externally).

The analysis of “Building BioData.pt” indicators using the OECD framework (Table 5) also corroborated the broad impact of the project, by retrieving indicators for all impact areas. In this case, the scientific impact produced by the activities of “Research Communities” assumes particular relevance. Nonetheless, as for the previous frameworks, the activities related to “Support to Research Communities” have a wider impact. The full dataset of this analysis can be consulted.¹¹

The indicators generated by “Industry and Entrepreneurship” activities could find matches or proxies in the lists of the three impact assessment approaches, with greater emphasis, as expected, in the Economic and Innovation impact areas, enriching the project with a translational dimension.

Overall, this analysis showed that the activities carried out to accomplish the “Building BioData.pt” overarching objectives of “Strengthening research, technological development and innovation” and “Enhancing research knowledge transfer from academy to industry” covered the broad range of impact areas foreseen for European Research Infrastructures. This not only puts this RI of the National Roadmap aligned with sibling international counterparts, but also suggests that BioData.pt is well aligned with ELIXIR, bringing to the national scientific system the state-of-the-art topics of computing, bioinformatics and data management, addressed by this ESFRI Landmark.

Finally, the relevance of the retrieved indicators was ascertained, considering the national context and the global environment of the BioData.pt mission. Arbitrary units (A.U.) were attributed to the alignment of each indicator with the objectives/goals of the BioData.pt (4.5 A.U.), ELIXIR (2 A.U.), National Roadmap of Research Infrastructures (2 A.U.), EOSC (1 A.U.) and the Sustainable Development Goals - SDG (0.5 A.U.), and each “Building BioData.pt” indicator was analysed with respect to its potential contribution to the objectives of these organisations/initiatives. In general, a relevant indicator for evaluating BioData.pt performance and impact needs to achieve the value of 6.5 A.U., thus embedding at least two parameters, the objectives of BioData.pt (4.5 A.U.) and either those of ELIXIR (2 A.U.) or the ones of the National Roadmap of Research Infrastructures (2 A.U.), organisations to which BioData.pt is bound by a formal relationship. Additional relevance of BioData.pt activities was observed by assessing their potential contribution to the EOSC (1 A.U.) and/or the SDG (0.5 A.U.) goals (Figure 3). The full dataset of this analysis can be consulted.¹²

Figure 3. Relevance of “Building BioData.pt” indicators identified in ESFRI (E), OECD (O) and RI-PATHS (R) impact assessment frameworks.

Arbitrary units (AU). Dashed line - threshold point.

Discussion

The RI-PATHS project listed 102 possible indicators to be considered when planning or carrying out performance and impact evaluation of RIs from different scientific areas (Health & Food, Data, Computing and Digital Research, Energy, Environment, Physical Sciences & Engineering, and Social & Cultural Innovation), and types (distributed or single-sited). RI-PATHS also proposes a set of pathways tailored to assess the socio-economic impact of specific project activities, encompassing smaller subgroups of indicators. For instance, pathways 5 - Learning and training by using RI facilities and services, 9 - Provision of specifically curated/edited data, and 11 - Creating and shaping scientific networks and communities seem better suited to evaluate the impact of BioData.pt training activities (e.g., “Ready for BioData Management?”), the CorkOak DB portal,¹³ and the “building and consolidation of our Communities and Platforms, in a multidisciplinary network to identify gaps in computing data management and bioinformatics for the life sciences and build solutions”.

From the referred universe of 102 RI-PATHS indicators, 25 were identified among those monitored during “Building BioData.pt”, which scatter in all impact areas, suggesting the broad socio-economic impact of BioData.pt. When selecting RI-PATHS pathways more appropriate to assess BioData.pt activities, the indicator retrieval ratio increases to ~50% of the corresponding indicators subgroup. This corroborates that the performance and impact indicators used to evaluate a specific project or activity need to be tailored to several parameters, namely the nature of the activity under assessment, e.g. a training program, a community, a data portal, and also a particular context such as a national RI, or a specific project, e.g., with a wider scope that covered simultaneously Enabling Science, Problem-Solving, and Science and Society, as in the case of the establishment of a national RI or to a particular activity with a narrower scope, in addition to the scientific domain or the entrepreneurial nature of such RI or activity.

Of the 21 indicators compiled by the ESFRI framework, 20 were monitored by “Building BioData.pt”. The missing indicator, Number of publicly available data sets used externally, relates to the optimisation of data use and is a relevant indicator for a RI like BioData.pt. Although BioData.pt has publicly available datasets (e.g., Pheno and CorkOakDB¹³), it does not currently maintain indicators to demonstrate how these are being used externally. This difficulty also intricates with the fact that there is still a long way to go in the formal acknowledgement of RI services by its users, which is harder when the goods provided are intangible and sometimes provided by third parties. In the future, this topic should be addressed by putting in place a tracking system. The ESFRI framework, currently with a very broad scope for each indicator, is more prone to embed the specificities of national RIs.

Finally, out of the 58 indicators compiled by the OECD framework, 32 were monitored by “Building BioData.pt”, 18 belonging to Scientific Impact and Training and Education, emphasizing the importance of these areas for BioData.pt. In this framework, there is a large collection of patent and industry related indicators that, for the time being, are not part of BioData.pt objectives. There is also a set of indicators proposed by OECD related to the European geography, that could be adapted to the national landscape (e.g. T24 – Regional firms funding RI).

It is noteworthy to remember that this exercise did not benefit from an ex-ante impact evaluation and therefore the indicators maintained were not previously selected to monitor future socio-economic impact but the accomplishment of the funders’ requirements (Portugal 2020 program and Fundação para a Ciência e a Tecnologia). Nonetheless, the three frameworks considered in this study were comprehensive enough to accommodate all indicators maintained during “Building BioData.pt”, in spite of the funding instruments, specific national context, nature of RI activities and role as a national node of a European RI.¹⁴ In addition, absent indicators may also prove useful to bring our attention to untracked activities that are relevant to the RI mission of an established national RI. One example is the long-term tracking of human resources in the RI, namely trainees, and scientific and managerial staff, their stays in the RI and their careers after leaving. The mechanism for such tracking is not trivial for a distributed organisation. Another example could be the already mentioned adaptation of indicators to monitor regional collaborations/networks and projects, or suppliers, and expenditure and income to the national scale. In the particular case of BioData.pt, which is now a legal entity, monitoring the number of its associates and the ability to capture users paid by the national funders, via time or service vouchers, would enable to assess the contribution of these activities to the RI sustainability.

The vast majority of the ESFRI, OECD and RI-PATHS indicators identified in “Building BioData.pt” surpassed the 6.5 A.U. in our scoring exercise, indicating that, this project was well aligned with current thinking around the performance and impact of RIs. This also reflects the natural integration of BioData.pt in the European and Portuguese research landscape. Furthermore, alignment with the EOSC goals was also very frequently observed underlining the commitment of BioData.pt with the best practices of FAIR data management and open science. It was interesting to observe the overlap between BioData.pt and the SDGs, suggesting that national RIs, as likely their European counterparts, can give a valuable contribution to global sustainable development, and are thus a public good. Below the threshold, three indicators contributing to the strategic objectives of the BioData.pt infrastructure were identified that did not promote the ELIXIR priorities or those of the National RoadMap of Research Infrastructures. These were: Number of employees, Income from commercial activities and number of entities paying for service, and Visitors on website and followers on social media/Outreach via the RI’s own web and social media. Although these indicators are not directly related to the mission of BioData.pt, they may assume a very relevant role for its long-term sustainability and thus, perhaps both the National Roadmap for Research Infrastructures and ELIXIR would like to consider this information when revising their priorities.

The process of collecting and categorising the indicators was well received by the project managers, who after a learning curve to apprehend the context and content of the new process, gained the understanding of the importance of evaluating the performance and impact of research infrastructure activities and the range of this potential impact. In detail, the project managers continuously collected and maintained the indicators generated by BioData.pt teams, which were reported annually to demonstrate the progress of the project. The participatory approach used to categorise the indicators, the B-MEM, allowed a wider participation of the infrastructure management staff in the conversion of a set of indicators in an impact analysis. The latter also enhanced the motivation and commitment of the managerial staff with the mission of BioData.pt. In a future opportunity, this process could be widen to the research/technical teams, enabling them to consider ex-ante the potential changes their activities will foster, and gather their contribution to the definition and collection of indicators. The wider vision gained by these teams is expected to create greater receptivity for the necessary indicator collection task, and also enhance their enthusiasm with their projects.

Final remarks

The monitoring of RIs performance and impact through indicators can be powerful in strategic management not only for continuous monitoring but also to inform future steps at the level of funders, optimization of processes and visibility by users and citizens.^15,16 Although highly qualified consultants are available to provide such service, national RIs are typically underfunded, and cannot support the cost. It is thus critical to establish simple and participatory methods to be implemented by the RI staff with oversight from the management team.

From this study, it is clear that the approaches developed by ESFRI, OECD and RI-PATHS are very useful and adaptable to the national context. It is also clear that special attention should be given to the national and thematic context of the activities under assessment. It is, thus, critical that each RI maintains the indicators that are more suitable to demonstrate the impact of its activities.

Overall, there is a strong alignment of “Building BioData.pt” with the mission of a research infrastructure as foreseen by ESFRI. The major flaw recognised under this study, respecting the difficulty to control external use and acknowledgement of RI assets, such us datasets, may need a collective solution to be addressed in the future.

This exercise was done backwards to provide lessons for the future, where ex ante the areas where impact aims to be achieved, the different types indicators for each activity to be collected, as well as the methods for capturing the indicators should be documented.

In this scope, the methodology used to assess the relevance of the indicators maintained during “Building BioData.pt” (Figure 3) can be useful to adapt the selection of indicators to a specific context, taking into account the mission and goals of the RI, and also those of national and international organizations and global initiatives composing the landscape where the RI develops its work and aims to have impact. To successfully assess performance and impact of RIs it is also very important to set up processes where the staff should be involved already from the planning stage, such as the BioData.pt monitoring and evaluation matrix (Figure 2). Such methodology permits the involvement of managerial and research/technical staff and incorporates both perspectives. Moreover, RI staff gains a wider view of the organisation and feels more committed with its success and motivated to contribute in the collection of the indicators in the long-term.

Data availability