[version 2; peer review: 2 approved, 1 approved with reservations]
All the authors contributed equally to this article.
Research leaders, policy makers and science strategists need evidence to support decision-making around research funding investment, policy and strategy. In recent years there has been a rapid expansion in the data sources available that shed light onto aspects of research quality, excellence, use, re-use and attention, and engagement. This is at a time when the modes and routes to share and communicate research findings and data are also changing.
In this opinion piece, we outline a series of considerations and interventions that are needed to ensure that research metric development is accompanied by appropriate scrutiny and governance, to properly support the needs of research assessors and decision-makers, while securing the confidence of the research community. Key among these are: agreed ‘gold standards’ around datasets and methodologies; full transparency around the calculation and derivation of research-related indicators; and a strategy and roadmap to take the discipline of scientific indicators and research assessment to a more robust and sustainable place.
We have responded to our reviewers' comments - particularly adding in reference to the issue of research indicators being potentially gamed. We have also made the description of the potential next steps to build the research metrics infrastructure more specific.
It is an exciting and challenging time for research evaluators and strategists; in the post-digital era, technical limitations around what can be used to assess different aspects of research are falling away. The availability of article-based citation metrics and indicators that capture research article reach, attention, and engagement is helping to reduce a reliance on misleading journal-based assumptions of scientific quality and importance. Many researchers now openly share components of their research – often within a research article, but increasingly outwith. For example, databases, datasets, software, and artistic outputs are often now on a range of platforms (e.g.
At their essence, research metrics are designed to shed light on a range of attributes of research to support decision-making around resource allocation and research funding strategy (including tenure, career appointments and grant applications). In addition, metrics today routinely support national research assessment exercises, as exemplified by
Policy experts and researchers have long petitioned to make research metrics more robust, evidence-based and
We are now at a pivotal point of the research indicator story where a political and administrative appetite for research metrics to build and sustain efficient and effective research systems co-exists with a burgeoning in the sources of intelligence about research outputs. What is needed to harness this momentum is cross-sector agreement on the next best steps and actions to make research metrics more robust, transparent and empowered to work for the whole research community. An important part of this is to make research indicators used, valued and acceptable to the research community for the purpose in which they have been designed. To date much of the debate around the challenge in using research metrics robustly has centred on the ease with which research-related metrics – and bibliometrics in particular – are gamed and suffer from Goodhart’s Law where “
Several initiatives are underway whose aim is, at least in part, to consider how to improve the evidence base upon which science is evaluated and make science more effective (see for example, the
Against this backdrop, we believe that there are now a number of very practical ingredients that can potentially act as part of a roadmap to ensure the development of robust and fair science indicators that have community support. We outline these below.
For research metrics to be understood and used consistently there needs to be agreement around common vocabulary and descriptors of terms. As an example,
The definitions themselves need to be definitive, openly sourced, managed, curated, versionable and quality assured. Additionally, the data upon which the indicator is best derived need to be identified. One of the challenges around research indicator derivation to date is that many of those in common usage are based upon opaque methodologies and proprietary datasets. This has eroded trust among the user base - many of whom don’t have access to the data - and pragmatically makes it difficult for particular metrics to be reproduced and explained.
An important concern around current research metrics is that they are often compiled and enabled through proprietary databases with locked access to the underlying data. This creates challenges for third parties wanting to replicate a metric, apply it in a different context or produce aggregate datasets from multiples sources. It also leads to mistrust and scepticism among users and those whose research is described (
The community needs a reference set – a Gold Standard (GS) dataset – for proper metrics development. A GS dataset would also enable an ongoing appraisal of best practice for a particular metric’s use and application – and potential inter-relationship with other metrics. Currently, a wide array of metrics is available. These make similar claims, but derive from different formulations. If enabled to work by correlating against a GS dataset, analysts can conduct systematic and rigorous testing and benchmarking for these options to surface the ones most useful across different applications. In short, while the open availability of raw metrics data is critical to transparency and to support innovation in metrics development and provisioning, we need a separate reference dataset that ensures the raw data which underlie a specific metric or metrics are properly preserved and audited.
In addition to the raw data, required analytical tools also need to be made available for true transparency and reproducibility (and thereby trust in the metrics). This includes products, such as a defined (minimum core) dataset, and open source standards on how the data are derived and defined (perhaps through an intermediary such as
Perhaps most importantly given the stakes involved, we need greater consensus around how science and research-related metrics are best used to support decision making in science. As noted earlier, metrics need to be created to answer specific research evaluation questions – and where possible be able to avoid the potential to be gamed. Research on research (science of science) is needed to help answer the important research evaluation questions and determine which metrics are useful and have the potential to provide insight to these research questions. As researchers adopt new ways to share and publish their research at speed, metrics and indicators that track and assess the value, quality and utility of those activities need to keep pace.
We see a valuable role for funders to play in supporting this particular research area. The community working in the field is small and funding can be difficult to allocate even where funding for research evaluation studies is available (such as the
Perhaps most importantly, it is paramount that and funding agencies and research institutions alike, work together to champion and incentivise the types of research and researcher behaviours that are likely to bring about desired outcomes and impacts – however wide the range of these might be. And, this might, interestingly, include de-emphasizing output in favour of seeking out more qualitative ways of assessing research (
As noted in
Currently, research and documentation on metrics is dispersed. As a non-disciplinary grouping, not a single scholarly community or society spans all the relevant groups working on theory, advancing analytics, data quality, visualisation, policy (and economics). No single party takes responsibility for collecting or documenting process, evidence of good or bad practice, or any other significant issues. The value of these resources may not be immediately obvious, but their absence can stunt the progress of metrics utility, innovation, transparency and dependability.
As researchers adopt new ways to share their scholarly contributions at speed, metrics which describe and provide insight into that work need to keep pace. Different metrics are likely to have different value across output types, research fields and in different circumstances. Yet we believe that a coordinated, cross community effort to enhance our knowledge and application of research metrics is both timely and sensible.
At the time of writing, we welcomed the work being initiated by the
To facilitate this, we recommend the establishment of a cross sector, community entity to be charged with building critical mass and momentum around research assessment and associated indicators/metrics. We envisage that there might be a number of guises that a cross community entity or effort could assume, including:
an
an
an
Such an entity could be governed and directed by a collective of independent research funding agencies or institutions, though would by necessity be a collective of relevant bodies. Or it could be configured entirely differently. What is important is that research assessment remains integral to the research enterprise; what is also clear is that as a discipline, it remains in its infancy and that to move forward, requires a cross-sector, cross research community involvement and engagement. We welcome initiatives that seek to seriously forge such collaboration, take research assessment to a more robust and sustainable footing, and as part of this, can help to spear-head the development, transparency and safe-guarding of ‘scientometrics’– be these quantitative or qualitative.
The authors have responded to the issues we raised in our earlier review. While the proposed next steps could use a great deal of fleshing out, they are also worthy of discussion by readers, so it would be good to allow this to be circulated to a wider audience.
Reviewer Expertise:
NA
We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
I believe that the authors have responded to the concerns raised appropriately and this should be indexed.
Reviewer Expertise:
NA
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
The opinions expressed in this paper are reasonable enough. My main reservation is that those opinions don’t go far beyond common sense, but that it is not at all obvious how to carry out these principles. As I said in my first review, preaching to scientists about what they should be doing is not very useful. Leading the way would be useful, but this paper does not do that.
The modified manuscript does nothing to address my earlier concerns. So, no, I would not amend my earlier recommendation. However, it is possible that my objection may be out-of-bounds, given the editorial policies of the journal with respect to this article type. If that is the case, then I have no further reservations about the manuscript.
Reviewer Expertise:
NA
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Thank you for the opportunity to review this manuscript. We should note, however, that given the nature of the manuscript, which is really an editorial, we are of the opinion that it probably does not require the kind of rigorous peer review usually applied to original research.
The manuscript presents a good summary of many of the issues facing research into metrics, and offers a plan for addressing them, at a very high level. We would suggest a few additions for improvement:
A nod to the fact that metrics can always be gamed, and that while making them more scientific could cut down on this risk, it will likely always be possible. It might also bear mentioning that no matter what metrics we end up with, they are no substitute for reading a particular paper. Put another way, metrics may be useful for certain things (eg large-scale productivity) but not others (eg quality). A more specific set of next steps. What first steps might funders, scientists, and administrators take? A systematic review and meta-analysis? A gathering to frame the questions and identify funding priorities? Etc.
Reviewer Expertise:
NA
We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.
I believe that the title is appropriate and the abstract captures the essence of this opinion piece. I’m emphasising that this is an opinion piece because judgement regarding sources and data would be very different. There is much that I like about this piece and one of the main things is that it puts out there a very important at present discussion: how we use indicators in research evaluation and how we can do this better (or at least in a way that doesn’t disadvantage the development of science). The authors are well informed about the state of play and have given serious consideration to what can be done.
I also can see how the very practical proposals in this piece could be implemented and yield some results.
My reservations are about the failure to reach beyond the ‘technical’ – this is very needed though probably outside of what the authors have set out to achieve here. This is why, I believe that this piece should be published and, possibly, scholars in the UK and beyond encouraged to take part in this kind of discussion.
Hope this helps.
Reviewer Expertise:
NA
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
In an opinion piece, Lin, Murphy, Taylor and Allen point out that: 1) researchers’ contributions to their fields are evaluated for purposes of research grant allocations, career advancement, prizes, etc.; and 2) the quality and validity of metrics that underlie these decisions are not well studied.
Lin and colleagues call for the development of a discipline that will improve the evidence and infrastructure with which science is evaluated. Their point is well-taken. The manuscript was useful (to me) in pointing out some references and links to initiatives that are now underway in this field. However, to tell the reader what needs to be done is much less useful than actually doing something. This manuscript offers some reasonable suggestions about steps that might improve the evaluation of science; the difficulty is that the article does not present any evidence of an advance. More than opinion is necessary to advance the field.
Reviewer Expertise:
NA
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.