Building the infrastructure to make science metrics more scientific

Introduction

It is an exciting and challenging time for research evaluators and strategists; in the post-digital era, technical limitations around what can be used to assess different aspects of research are falling away. The availability of article-based citation metrics and indicators that capture research article reach, attention, and engagement is helping to reduce a reliance on misleading journal-based assumptions of scientific quality and importance. Many researchers now openly share components of their research – often within a research article, but increasingly outwith. For example, databases, datasets, software, and artistic outputs are often now on a range of platforms (e.g. Figshare, Zenodo) and independently citable (through the use of a digital identifier, such as a DOI). In addition, many researchers share analysis through non-traditional media (e.g. preprints, blog posts and policy documents).

At their essence, research metrics are designed to shed light on a range of attributes of research to support decision-making around resource allocation and research funding strategy (including tenure, career appointments and grant applications). In addition, metrics today routinely support national research assessment exercises, as exemplified by REF2014 in the UK and ERC2015 in Australia. Despite this, there continues to be limited investment in either research on the quality and validity of the indicators or the governance and stewardship of the data upon which indicators are derived.

Policy experts and researchers have long petitioned to make research metrics more robust, evidence-based and scientific (Lane, 2010) and therefore acceptable to the community they are meant to serve. Recent analyses have also reported on the current limitations of research metrics, calling for more research on, and improvements in, the infrastructure to support science indicators (Hicks et al., 2015; Wilsdon et al., 2015). The EU also recently issued a consultation to put ‘alternative’ metrics on firmer footing as part of its drive to encourage open science approaches and robust ways to evaluate research (Amsterdam Call for Action on Open Science, 2016). However, the ‘science’ of research metrics (scientometrics) paradoxically remains an orphan discipline given that more effective and accurate science metrics could make science more effective.

Building an evidence base for metrics

We are now at a pivotal point of the research indicator story where a political and administrative appetite for research metrics to build and sustain efficient and effective research systems co-exists with a burgeoning in the sources of intelligence about research outputs. What is needed to harness this momentum is cross-sector agreement on the next best steps and actions to make research metrics more robust, transparent and empowered to work for the whole research community. An important part of this is to make research indicators used, valued and acceptable to the research community for the purpose in which they have been designed. To date much of the debate around the challenge in using research metrics robustly has centred on the ease with which research-related metrics – and bibliometrics in particular – are gamed and suffer from Goodhart’s Law where “when a measure becomes a target, it ceases to be a good measure” (Elton, 2004) so that they become poor indicators of either productivity or research quality.

Several initiatives are underway whose aim is, at least in part, to consider how to improve the evidence base upon which science is evaluated and make science more effective (see for example, the EU Open Science Policy Platform, and the UK Forum for Responsible Research Metrics [announced in September 2016]). The key ways that such initiatives will be able to make a real difference, is four-fold. First, ensure active participation from across the whole scientific research community in a broad way to include researchers, institutions and funding agencies, alongside scientific publishers, learned societies and technology platform providers. Second, deliver a roadmap for the key requirements needed to build and assure quality science metrics for the benefit of science. This should include consideration of how research productivity is best tracked and assessed, both quantitatively and qualitatively. Third, question existing assumptions around how we conduct and reward research, and test out new approaches and ways of working – and again considering how to incentivise the type of research that is required to deliver the required goals. Fourth, secure access to resources and influence, as well as make actionable decisions.

Against this backdrop, we believe that there are now a number of very practical ingredients that can potentially act as part of a roadmap to ensure the development of robust and fair science indicators that have community support. We outline these below.

Moving forward: a path to fulfil these needs

As researchers adopt new ways to share their scholarly contributions at speed, metrics which describe and provide insight into that work need to keep pace. Different metrics are likely to have different value across output types, research fields and in different circumstances. Yet we believe that a coordinated, cross community effort to enhance our knowledge and application of research metrics is both timely and sensible.

At the time of writing, we welcomed the work being initiated by the EU Consultation on Metrics and the announcement of the UK’s Responsible Metrics Forum, both which aim to some degree to rethink the scope and use of research metrics. However, we would like to see the discourse move far beyond descriptors of the challenges of using current research metrics responsibly, to one that helps the research community to build research assessment into a discipline that can actively support efficiency in science and research. And one that starts to take practical steps to build the infrastructure to support research assessment and develop indicators.

To facilitate this, we recommend the establishment of a cross sector, community entity to be charged with building critical mass and momentum around research assessment and associated indicators/metrics. We envisage that there might be a number of guises that a cross community entity or effort could assume, including:

Such an entity could be governed and directed by a collective of independent research funding agencies or institutions, though would by necessity be a collective of relevant bodies. Or it could be configured entirely differently. What is important is that research assessment remains integral to the research enterprise; what is also clear is that as a discipline, it remains in its infancy and that to move forward, requires a cross-sector, cross research community involvement and engagement. We welcome initiatives that seek to seriously forge such collaboration, take research assessment to a more robust and sustainable footing, and as part of this, can help to spear-head the development, transparency and safe-guarding of ‘scientometrics’– be these quantitative or qualitative.