‘The great publication race’ vs ‘abandon paper counting’: Benchmarking ECR publication and co-authorship rates over past 50 years to inform research evaluation

Dave Kendal; Kate E. Lee; Kylie Soanes; Caragh G. Threlfall

doi:10.12688/f1000research.75604.1

Home Browse ‘The great publication race’ vs ‘abandon paper counting’: Benchmarking...

ALL Metrics

-

Views

-

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

‘The great publication race’ vs ‘abandon paper counting’: Benchmarking ECR publication and co-authorship rates over past 50 years to inform research evaluation

[version 1; peer review: 2 approved with reservations]

Dave Kendal ¹, Kate E. Lee², Kylie Soanes², Caragh G. Threlfall³

PUBLISHED 26 Jan 2022

Author details Author details

¹ Healthy Landscapes Research Group, School of Geography, Planning and Spatial Sciences, University of Tasmania, Hobart, Tas, 7000, Australia
² School of Ecosystem and Forest Sciences, The University of Melbourne, Parkville, Vic, 3010, Australia
³ School of Life and Environmental Sciences, The University of Sydney, Sydney, NSW, 2006, Australia

Dave Kendal
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Kate E. Lee
Roles: Conceptualization, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing

Kylie Soanes
Roles: Conceptualization, Investigation, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Caragh G. Threlfall
Roles: Investigation, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

Abstract

Background: Publication and co-authorship rates have been increasing over decades. In response, calls are being made to restrict the number of publications included in research evaluations. Yet there is little evidence to guide publication expectations and inform research evaluation for early career researchers (ECRs).

Methods: Here we examine the early career publication and co-authorship records between 1970 and 2019 of >140,000 authors of 2.8 million publications, to identify how publication and co-authorship rates have changed over the last 50 years. This examination is conducted in order to develop benchmarks of median publication rates for sensibly evaluating ECR research productivity, and to explore success in meeting these benchmarks with different co-authorship strategies using regression models.

Results: Publication rates of multidisciplinary ECRs publishing in Nature, Science and PNAS have increased by 46% over the last 50 years and that publications rates in a set of disciplinary journals have increased by 105%. Co-authorship rates have increased even more, particularly for the multidisciplinary sample which now has 572% more co-authors per publication. Benchmarks based on median publication rates for all authors increased from one publication per year at the start of a career, to four publications per year after 10 years of publishing, and one first-author publication across all years. The probability of meeting these benchmarks increases when authors publish with different co-authors, and first authorship rates decrease for ECRs with many co-authors per publication.

Conclusion: This evidence could be used to inform sensible publishing expectations for ECRs and the institutions they work for, and to inform calls to limit the number of publications produced by researchers and those used in research evaluations.

Keywords

publication metrics, publish or perish, academic careers, early-career researchers, postdocs, ECRs

Corresponding author: Dave Kendal

Competing interests: No competing interests were disclosed.

Grant information: This work was supported by the Clean Air and Urban Landscapes hub (DK, KL, KS and CT) and the Threatened Species Recovery hub (KS), funded by the Australian Government’s National Environmental Science Program.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2022 Kendal D et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Kendal D, Lee KE, Soanes K and Threlfall CG. ‘The great publication race’ vs ‘abandon paper counting’: Benchmarking ECR publication and co-authorship rates over past 50 years to inform research evaluation [version 1; peer review: 2 approved with reservations]. F1000Research 2022, 11:95 (https://doi.org/10.12688/f1000research.75604.1) First published: 26 Jan 2022, 11:95 (https://doi.org/10.12688/f1000research.75604.1) Latest published: 26 Jan 2022, 11:95 (https://doi.org/10.12688/f1000research.75604.1)

Introduction

Publication metrics are commonly used for managing academic expectations and evaluating research performance, both by researchers and the institutions they work for.¹ Despite widespread concerns about their use and abuse,¹^–³ publication counts, citations and impact factors are still commonly used to guide academic hiring and promotion decisions, and to allocate funding resources.⁴^–⁶ Yet there is increasing concern that a focus on such metrics has negative consequences, both for science and the researchers themselves. An unbalanced focus on volume may come at the expense of other important aspects of scientific endeavour, such as quality, engagement, and impact. A reward system that incentivises publication quantity can reduce research quality⁷^,⁸ as the focus shifts to researcher status rather than knowledge gain,¹ leading to “unsustainable science”.⁹ In response, calls are being made to “abandon paper counting”,³ limit the number of publications used in research performance evaluations³ and even limit the number of papers published by researchers.¹⁰ Yet this debate about increasing publication rates, and an appropriate response by researchers and those who evaluate them, has largely ignored early career researchers (ECRs).

There is potential that this unsustainable increase in publication rates could be amplified amongst ECRs, particularly as commonly used publication metrics do not fairly describe their performance; forcing ECRs to focus on publication counting. For example, the widely-used h-index is poorly suited to evaluating early-career research¹¹^–¹³ and it can take many years for high quality research to be reflected in an increase in an author’s h-index. Predictors of academic ‘success’ are often difficult for ECRs to control; publishing earlier,¹⁴^–¹⁶ working at a more prestigious institution,⁵ or publishing with top scientists¹⁷ are outside of the control of most ECRs. In the context of uncapped publication expectations, the primary goal for many ECRs is to publish as many papers as possible: “Four papers are better than three. And five are better than four”.¹⁸ However, favouring quantity of publications is a potentially maladaptive behaviour. It is a poor predictor of later research ‘quality’ as measured by indicators such as citation rates.¹⁹ Research evaluation that defaults to publication counting will also disadvantage ECRs with families, those who teach, those who engage with industry or the broader public, or those who work in smaller labs; as they balance producing publications with a variety of competing responsibilities.²⁰^,²¹

There is currently little empirical data to make sense of these debates for ECRs, or for the research institutions and grant assessors evaluating their research performance. We know that ECRs are publishing earlier compared with ECRs from previous decades, often before their PhD is complete.²² Yet it is not clear that the pressure to publish is actually leading to increased publication rates for current ECRs, compared with previous generations. Publication strategies that could be potentially useful for ECRs and those who evaluate their research performance, have yet to be tested for ECRs. For example, strategies such as publishing with a large number of co-authors¹⁴^,²³ or in a range of different titles.²³^,²⁴ Calls to severely restrict publishing or the number of publications used in research evaluation³^,¹⁰ may diminish the pressure to publish, but could be counterproductive if applied to ECRs who are busily developing networks and skills through publications.

A research evaluation framework that is specific to the early years of a career is needed. This should be based on an understanding of how ECR publication and co-authorship rates have changed through time, how they progress during a career, and what strategies best support research ‘success’. It should remove the pressures generated by uncapped publication expectations, and at the same time it should not unnecessarily restrict publication activity, which could inhibit collaboration and skill development. It should recognise the changing patterns of co-authorship, from large team-based publications to sole authorship. This study aims to generate empirical data to support the development of a research evaluation framework suitable for ECRs by:

1) Characterising how publication rates for ECRs have changed over the last 50 years.
2) Developing evidence-based benchmarks for ECR publication rates as careers develop.
3) Exploring the effect of co-authorship strategies on the probability of researchers meeting these benchmarks.

Methods

Author samples

Two author samples were selected. The first (multidisciplinary) sample was selected to gain insights into publishing and co-authorship trends between 1970-2019 in the broader field of science, particularly high-ranking international journals that are so often used as an indicator of career success. This included all authors who published in the leading, multidisciplinary scientific journals Nature, Science and PNAS in 2019. A second (disciplinary) sample was selected to explore publication and co-authorship rates of ECRs who routinely publish in discipline-specific journals. This sample was also to compare whether trends observed in the publication rates of leading scientists, are also seen in a cohort of researchers exclusively publishing in more specialised journals – perhaps a more familiar experience for most ECRs. The disciplinary sample included all authors who in 2019 published in any journal that the authors of this article had published in at least twice, including a wide range of ecology, interdisciplinary landscape, and environmental psychology journals (n=21, Table 1).

Table 1. Journals used in the disciplinary author sample.

Journal	SJR 2018	H index
Frontiers in Ecology and the Environment	4.5	135
Global Ecology and Biogeography	3.5	127
Conservation Biology	3	201
Journal of Applied Ecology	2.7	159
Biological Conservation	2.2	173
Journal of Environmental Psychology	2	112
Environmental Science and Policy	1.9	95
Landscape and Urban Planning	1.8	132
Science of the Total Environment	1.5	205
Sustainability Science	1.4	42
Ecology and Evolution	1.3	47
Journal of Environmental Management	1.2	146
Biodiversity and Conservation	1.2	114
PLoS ONE	1.1	268
Urban Forestry and Urban Greening	1	58
Urban Ecosystems	0.9	44
Austral Ecology	0.8	77
Landscape Research	0.6	36
Australian Mammalogy	0.5	22
Acta Horticulturae	0.2	51
Victorian Naturalist	0.2	8

Data collection

The ECR publication history of these two author samples were collected using the Scopus API, accessed using the rscopus package in R v3.6 in January 2020. A list of all 2019 publications with a Scopus type of ‘Article’ or ‘Review’ (and ‘Letter’ in Nature) were retrieved for each journal of interest using the scopus_search function based on the ISSN of each publication.⁴⁵ All authors for each article were retrieved.

There is no consensus on the definition of ‘early career’. Previous studies have defined ECRs as researchers within five to twelve years of their first year of publishing.¹⁶^,²⁵ In a study of research productivity by career stage, ECRs had an average of 4.5 year’s experience post-PhD, compared with 14.1 years for mid-career faculty members.²⁶ In this study we defined ‘early career’ as the first ten years of a career to encompass a broad range of definitions – subsets of the data presented here can be applied to shorter definitions of ECR career length.

‘Career age’ or ‘academic age’ are often defined as the number of years since first publication,¹¹^,¹³ or years since PhD.²⁷ However, consistent with other studies,²⁸ we observed that many researchers in our author cohorts had gaps in their early years of publishing, possibly due to career gaps [e.g. due to parental leave, illness or caring responsibilities] or breaks between completing honours or masters degrees and beginning PhD studies along with the increasing prevalence of publishing before completing a PhD.²² As early career publishing rates are sensitive to these gaps, we controlled for them by calculating the career age as the number of ‘active’ years of publishing, i.e., excluding years with no publications. While recognising that continuous publishing is an important predictor of academic excellence,²⁸ excluding these career gaps better reflects the publishing experience of most ECRs.

The early career publication history (i.e. the first ten active years) of n=55,332 authors for the multidisciplinary sample and n=85,793 authors for the disciplinary sample was retrieved using the author_retrieval function of the rscopus package. Details of articles including publication date, publication title, and co-authors, and author details including institution, city, and country were extracted. Erratum and corrigendum were removed from the analyses. For each published article (n= 1,195,246 articles for the multidisciplinary sample and n= 1,597,069 articles for the disciplinary sample), the author position, and number of co-authors were determined.

Analyses

For each active year of publication for each author, the number of publications, number of first author publications, total number of unique co-authors, and the average number of co-authors per publication was calculated. Adopting the approach of Calver et al., we drew on ecological statistics, using the Jaccard index to calculate co-author dissimilarity across publications using the vegdist function in the vegan package in R.²⁹ This measure of dissimilarity will approach 1 when co-authors vary across publications, and approach 0 when co-authors are similar across all publications, in any given year.

A generalised linear model using a Poisson distribution was used to predict publication rates (total number of publications, and number of first author publications) and a log model was used to predict co-authorship rates (average co-authors per publication) and a Gaussian distribution to predict co-author dissimilarity across publications, from the first year of publication and career age.

In the context of increasing concern over the focus on quantity over quality in academic publishing and research evaluation,³ we explored alternative metrics that could provide benchmarks for ECR publication rates using the multidisciplinary cohort. We calculated the median and mean number of publications for all multidisciplinary authors for each active career year, and the median number of first author publications of all multidisciplinary authors across all active years. We also provide comparison publication rates for authors from different co-authorship environments: authors in the top quartile of average number of co-authors over the last decade and for authors from the bottom quartile of average number of co-authors over the last decade.

Based on these data, we determined measures of publishing success that limit publication expectations using three metrics: did the author publish at least the: 1) median, 2) mean number of publications for their career age in any given year, and 3) did the author publish at least the median number of first-author publications in any given year? A set of logistic models was developed to predict the probability of meeting these metrics from the average number of co-authors per publication (log transformed), and the dissimilarity of authors across publications (as measured by the Jaccard index).

Results

Are ECR publication rates changing over time?

ECR publication rates have increased over the last 50 years, particularly in our disciplinary sample which is now publishing at higher rates than authors in Nature, Science and PNAS. Multidisciplinary ECR publication rates have increased 46% between 1970 and 2019 (Figure 1A). Current ECRs are publishing an average of 3.5 publications/year after five active years, compared with 2.4 publications/year after five years for ECRs in 1970 (Figure 1A). This increase is dwarfed by the increases in publication rates that occur over the first ten years of a career. Current ECRs publication rates increase threefold over the first ten years of publishing, from 2.1 publications/year in the first year of publishing to 6.6 publications/year in the tenth year of publishing. There were contrasting findings for first author publication rates, which have declined 60% over the last 50 years. After five active years of publishing, authors in 1970 were publishing 1.5 first author publications per year, while in 2019 this figure was 0.6 first author publications per year (Figure 1B). There was a smaller increase in first author publication rates over the first ten years of a career, which increase 31% between the first and tenth active year of publishing for current ECRs (Figure 1D).

Figure 1. Prediction of publications per year (A and C) and first author publications (B and D) by first year of publication for our multidisciplinary author sample (A and B) and our disciplinary sample (C and D).

The increase in publication rates over time was much more dramatic in our disciplinary sample, increasing 105% between 1970 and 2019, from rates that were substantially lower than the multidisciplinary sample to rates that are now higher. After five active years of publishing, rates had increased from 1.7 publications/year in 1970 to 3.6 publications/year in 2019 and from 3.4 to 7.1 publications/year after ten years (Figure 1D). There was a smaller decline of 28% in first author publications per year, from 1.3 in 1970 to 0.9 in 2019 after ten years of publishing.

There were substantial increases in average co-authorship rates over the last 50 years, particularly amongst multidisciplinary ECRs (Figure 2A), who are now publishing with 572% more co-authors per publication compared with 50 years ago. The average number of co-authors per publication in the tenth career year increased from an average of 2.9 authors per publication in 1970 to 16.6 in 2019. In our disciplinary sample, this increased 230% from 2.9 co-authors per publication in 1970 to 9.5 co-authors per publication in 2019 in the tenth career year. There were also increases in co-authorship rates over the first ten years of publishing of 61% in the multidisciplinary sample and 49% in the disciplinary sample. Dissimilarity of authors across publications has increased slightly from 1970 to 2019 in the multidisciplinary and disciplinary samples. However, dissimilarity increased substantially over the first ten years of a career in both samples (Figure 2B and Figure 2D).

Figure 2. Prediction of average co-authors per year (A and C) and author dissimilarity (B and D) by first year of publication for our multidisciplinary author sample (A and B) and our disciplinary sample (C and D).

Our new measure of active year (years since first publication, excluding years with no publications) was similar to career age (years since first publication; Figure 3A). One quarter of authors had no publication gaps in their first ten years of publishing, while 58% of authors had three or fewer years with no publications (Figure 3B). While the differences between active year and career age may be relatively small, active years should more accurately describe the opportunity of ECRs who have had career interruptions or a break between publishing from honours or master’s studies, than career age or number of years since PhD completion.

Figure 3. A – Mean publications per year by active year (solid circle) and career age (open square) for all authors. B – The proportion (bars) and cumulative proportion (circles) of authors with no publications in some years over the first 10 active years of their career.

Evidence-based benchmarks for ECR publication rates

Our selected benchmarks for evaluation were the career-age controlled median and mean publication rates of 55,332 authors publishing in Nature, Science and PNAS in 2019 (Table 2, Figure 4), and publishing at least the median number of first-author publications per year relative to their cohort. Median publication rates increased from one publication/year in the first active year of publishing, to two in the fifth year of publishing to four in the tenth year of publishing (Table 2). Mean publication rates increased due to a number of authors with substantially higher publication rates (as seen by the outliers in Figure 4), from 1.6 publications/year in the first active year of publishing, to 3.3 in the fifth year of publishing to 5.2 in the tenth year of publishing (Table 2). Median and mean publication rates were somewhat higher (34%) for authors with many co-authors, and slightly lower (13%) for authors with few co-authors. The median first author publication rate was one first author publication/year in the first ten years of publishing. In any given year 71% of multidisciplinary authors met the median publication rate benchmark, 33% met the mean publication rate benchmark, and 53% met the first author publication benchmark.

Table 2. Mean and median publications rates (publications/year) at different career ages for 55,332 authors who published in Nature, Science and PNAS in 2019.

Rates for authors with many co-authors (top quartile>13.1 average co-authors) and few co-authors (bottom quartile<5 average co-authors) are shown for comparison.

			Low [Q1] #coauthors		High [Q4] #coauthors
Career age	Mean	Median	Mean	Q1-med	Q4-mean	Q4-med
1	1.6	1	1.5	1	1.8	1
2	2.1	1	1.9	1	2.9	2
3	2.6	2	2.3	2	3.5	2
4	3.0	2	2.6	2	4.2	3
5	3.3	2	2.8	2	4.7	3
6	3.6	3	3.1	2	5.1	3
7	4.0	3	3.4	2	5.4	3
8	4.3	3	3.7	3	5.8	4
9	4.8	3	4.0	3	6.2	4
10	5.2	4	4.4	3	6.7	4

Figure 4. A boxplot of publications per year by active years of publishing.

Median values are shown as thick horizontal lines, mean values as solid circles, and the box shows the interquartile range.

The effect of co-authorship strategies on the probability of researchers meeting these benchmarks

The likelihood of meeting these benchmarks was higher for ECRs whose co-authors vary across publications each year (Figure 5, Table 3) as measured by co-author dissimilarity (Jaccard index). This is not surprising given that publishing with different authors necessarily means publishing more. Increasing the number of co-authors per publication also had positive effect (although smaller than author dissimilarity) on the likelihood of achieving median and mean publication rates (Figure 5A, C, Table 3). However, publishing with a greater number of co-authors per publication greatly decreased the probability of having a first-author publications (Figure 5F, Table 3).

Figure 5. Probability of meeting median publication rates (A and B), mean publication rates (C and D), and publishing at least one first author paper (E and F) in any given year based on mean number of co-authors per publication (A, C and E) and the dissimilarity of co-authors across publications within the year as measured by the Jaccard index (B, D and E).

95% confidence intervals are shown.

Table 3. Model fit statistics.

Discussion

Current ECRs have higher rates of publication than ECRs from earlier decades. This is true for scientists publishing in leading multidisciplinary journals, and much more so for a cohort of ECRs publishing in disciplinary journals spanning ecology, interdisciplinary landscape, and environmental psychology. In the context of these findings, and heeding the call to “abandon paper counting”,³ we present three evidence-based benchmarks of research performance for ECRs: the median and mean yearly publication rates, and median yearly first-author publication rates, relative to their disciplinary cohort. The best predictor of meeting these publication benchmarks was publishing with co-authors that vary from publication to publication. Perhaps surprisingly, having more co-authors per publication was negatively related to meeting the first-authorship benchmark.

Changes in ECR publication rates over time

The acceleration in publication rates observed in our study is unlikely to be sustainable. While there are undoubtedly ECRs who publish at the higher rates observed over the last decade while maintaining research quality and impact (and quality of life), this is unlikely to be true for many researchers. In order to develop a competitive track record, ECRs are often advised to work long hours, avoid activities that aren’t directly linked to career advancement,³⁰ and write without the distractions of work colleagues, friends and families.³¹^,³² These effects may be compounded by short-term contracts which demand continual output at the expense of “human and social capital accumulation”, and can lead to career termination due to unavoidable fluctuations in output.³³ An unbalanced focus on publication rates risks compromising other aspects of scientific endeavour and academic development, and it may lead to declines in research quality and impact, compromise work-life balance and mental health, or reduce diversity as disadvantaged researchers slip from the system.¹⁸^,³⁰^,³⁴

Interestingly, rates of first authorship are decreasing, particularly in the multidisciplinary author group. This seems a natural corollary to the even greater increases observed in co-authorship rates over the last 50 years, as has previously been well demonstrated for established researchers,³⁵ and may also be partly offset by increases in joint first-authorship publications.³⁶ While the average number of co-authors increases slightly over the first ten publishing years of an academic career, there has been a substantial increase over the last 50 years for all ECRs. This demonstrates a substantial change in publishing patterns, away from individual and small group collaborations to larger team collaborations. Relatively stable patterns in the dissimilarity of co-authors across publications highlight that this shift is to relatively stable larger teams of authors rather than larger individual networks of co-authors.

Concerningly, the real size of the observed acceleration in publication rates is likely to be greater than described here. Our data is likely to be oversampling the most successful established scientists, as ECR’s from earlier decades who ceased publishing before 2019 are not included. This is consistent with previous research which has identified the attrition of less-productive researchers as a factor in assessing publication rates over time.²⁷

Evidence-based benchmarks for ECR publication rates

There is a clear need for evaluation strategies that are tailored to variations in publication rates over the early years of a career, and do not further encourage unsustainable rates of publication – that avoid assuming ever increasing quantities of publications are better.¹⁸ We have proposed a suite of evidence-based benchmarks with which to set ECR publishing expectations, and to evaluate ECR publication performance. These benchmarks incorporate career age, ensuring that they more appropriately and fairly assess performance of those early in their career. A suite of metrics allows for a more comprehensive view of research productivity (e.g., volume, collaboration, and research leadership), while still permitting ‘outstanding’ performance to be identified. For example, there is a substantial difference between median and mean publication rates as a result of enormous variation in the output of individual researchers, with some ECRs in this sample publishing more than 100 papers in some years. This difference is potentially useful for evaluating ECR productivity as the proportion of researchers meeting mean publication rate benchmarks was half the rate of those meeting median publication rates. This allows some discrimination between ECRs based on publication productivity, while still limiting the expectations on the numbers of publications used in evaluations. ECRs and institutions could use these to set expectations for sensible rates of publishing or to choose the number of publications to use in research evaluation.³

Care must be taken in the use of this evidence base. It does not control for part-time work and the fraction of employment devoted research. While it does control for career gaps spanning calendar years, it does not control for shorter career gaps or periods of reduced activity due to parenting, caring responsibilities, health and other pressures that can substantially affect research productivity. It is likely that there is some variation in publication rates across different research areas, and future research may develop subject-specific benchmarks. Further research could explore the role of different kinds of publications such as books that are important in some disciplines.²⁶

The effect of co-authorship strategies on the probability of researchers meeting these benchmarks

The finding that publishing with dissimilar authors across publications is a better predictor of meeting these benchmarks than large numbers of co-authors per paper, is consistent with studies of academic success in established researchers that highlight the benefits of collaboration, and the pitfalls of over-reliance on publications with many authors.²⁴^,³³^,³⁷^,³⁸ Previous studies have shown that publishing with a moderate number of co-authors is associated with higher publication and citation rates.³⁹ Further afield, international collaboration is associated with increased publication productivity.⁴⁰^,⁴¹

The strategy of developing a broad and varied network of co-authors may also lead to a range of other benefits such as grant writing opportunities, exposure to diverse views and development of professional networks, and consequently job opportunities. Research suggests that opportunities for networking and collaboration can lead to an increased perception of quality of a research environment and improved publication rates for ECRs.⁴² More broadly, increased networking can lead to perceived and actual benefits in the workplace,⁴³ through mechanisms such as increased access to information, resources and mentoring.⁴⁴

Study limitations

The multidisciplinary author sample used here have all published in top journals that most researchers do not publish in. Thus, basing benchmarks on the ECR track record of these authors may not be fair to all ECRs. Nonetheless, the comparison with a disciplinary author sample that has largely converged with the publishing rates of these authors provides some confidence that the benchmarks will be valid, at least in some science disciplines. Further research could help to better understand which disciplines these benchmarks are indeed valid for, and which disciplines require alternative benchmarks (or for which publishing benchmarks may not be appropriate).

While Scopus historical data is likely to be limited and may not include all publications of ECRs from the 1970s and 1980s, this limitation is balanced to an extent by our sample only including authors who continue to publish in top journals in 2019– and this group is likely to have higher publication rates than the average ECR from the 1970s and 1980s.

Conclusion

There is growing concern about the effects of the ‘publish or perish’ mentality on the career prospects and wellbeing of young researchers.¹⁸^,²⁰ In response, some authors are arguing that the number of publications included in research evaluations should be limited.³ Our findings show that current early career researchers are indeed publishing at an accelerated rate compared with their peers from previous decades. This is more pronounced for our disciplinary peers compared to authors in leading multidisciplinary journals. This work is a first step in introducing an evidence base to set publishing expectations and support the current debate around limiting publication rates and the number of publications used in research evaluations for ECRs. Reducing the pressure to always be publishing more, may allow ECRs to focus on the broader societal and institutional responsibilities that come with a career as a researcher, while still demonstrating research excellence. This may encourage ECRs to invest time in other endeavours leading to personal, academic, and societal benefits, such as grant applications, student supervision, collaboration, teaching, communication, outreach, industry engagement and academic service. These are critical pillars of an academic career. Framed in this way, developing collaborative networks with other researchers and research partners becomes a useful strategy for success rather than a distraction from writing more publications. These findings do not diminish the need for qualitative, place and mission specific assessments of research performance,² but instead provide a suite of metrics that can more reasonably guide and evaluate the publication performance of early career researchers.

Data availability

Harvard Dataverse: ECR publication metrics https://doi.org/10.7910/DVN/SM9KMI⁴⁵

This project contains the following underlying data:

• pubs_year-metadata-Jan2021.tab
• pubs_year-multi.csv
• pubs_year-us.csv

Data are available under the terms of the Creative Commons Zero “No rights reserved” data waiver (CC0 1.0 Public domain dedication).

Acknowledgements

We would like to thank Colin Simpfendorfer and Helene Marsh for comments on an earlier version of this manuscript.

References

1. Chapman CA, Bicca-Marques JC, Calvignac-Spencer S, et al.: Games academics play and their consequences: How authorship, h-index and journal impact factors are shaping the future of academia. Proc. R. Soc. B Biol. Sci. 2019; 286(1916). Publisher Full Text
2. Hicks D, Wouters P, Waltman L, et al.: The Leiden Manifesto for research metrics. Use these ten principles to guide research evaluation. Nature. 2015; 520(7548): 9–11. Publisher Full Text
3. Finkel A: The road to bad research is paved with good intentions. Nature. 2019; 566(7744): 297. PubMed Abstract | Publisher Full Text
4. Abbott A, Cyranoski D, Jones N, et al.: Metrics: Do metrics matter?. Nature. 2010; 465(7300): 860–862. Publisher Full Text
5. Way SF, Morgan AC, Clauset A, et al.: The misleading narrative of the canonical faculty productivity trajectory. Proc. Natl. Acad. Sci. U. S. A. 2017; 114(44): E9216–E9223. PubMed Abstract | Publisher Full Text
6. Tregoning J: How will you judge me if not by impact factor?. Nature. 2018; 558: 345. PubMed Abstract | Publisher Full Text
7. Schneider JW, Aagaard K, Bloch CW: What happens when national research funding is linked to differentiated publication counts? A comparison of the Australian and Norwegian publication-based funding models. Res. Eval. 2016; 25(3): 244–256. Publisher Full Text
8. Butler L: Modifying publication practices in response to funding formulas. Res. Eval. 2003; 12(1): 39–46. Publisher Full Text
9. Paasche Ø, Österblom H: Unsustainable Science. One Earth. 2019; 1(1): 39–42. Publisher Full Text
10. Frith U: Fast Lane to Slow Science. Trends Cogn. Sci. 2020; 24(1): 1–2. PubMed Abstract | Publisher Full Text
11. Carbon CC: The carbon_h-factor: Predicting individuals’ research impact at early stages of their career. PLoS One. 2011; 6(12). PubMed Abstract | Publisher Full Text
12. Gruber T: Academic sell-out: how an obsession with metrics and rankings is damaging academia. J. Mark. High. Educ. 2014; 24(2): 165–177. Publisher Full Text
13. Ayaz S, Masood N, Islam MA: Predicting scientific impact based on h-index. Scientometrics. 2018; 114(3): 993–1010. Publisher Full Text
14. Laurance WF, Useche DC, Laurance SG, et al.: Predicting Publication Success for Biologists. Bioscience. 2013; 63(10): 817–823. Publisher Full Text
15. Zeng A, Shen Z, Zhou J, et al.: The science of science: From the perspective of complex systems. Phys. Rep. 2017; 714–715: 1–73. Publisher Full Text
16. Zhang Y, Yu Q: What is the best article publishing strategy for early career scientists?. Scientometrics. 2020; 122(1): 397–408. Publisher Full Text
17. Li W, Aste T, Caccioli F, et al.: Early coauthorship with top scientists predicts success in academic careers. Nat. Commun. 2019; 10(1): 1–9. Publisher Full Text
18. Powell K: Young, talented and fed-up: Scientists tell their stories. Nature. 2016; 538(7626): 446–449. PubMed Abstract | Publisher Full Text
19. Bornmann L, Williams R: Can the journal impact factor be used as a criterion for the selection of junior researchers? A large-scale empirical study based on ResearcherID data. J. Informet. 2017; 11(3): 788–799. Publisher Full Text
20. Shen H: Plight of the postdoc. Nature. 2015; 525: 279–281. PubMed Abstract | Publisher Full Text
21. Small GE: The Post-doc dilemma. Nature. 2012; 483: 235. Publisher Full Text
22. Waaijer CJF, Macaluso B, Sugimoto CR, et al.: Stability and longevity in the publication careers of U.S. doctorate recipients. PLoS One. 2016; 11(4): 1–15. Publisher Full Text
23. Petersen AM, Jung WS, Yang JS, et al.: Quantitative and empirical demonstration of the Matthew effect in a study of career longevity. Proc. Natl. Acad. Sci. U. S. A. 2011; 108(1): 18–23. PubMed Abstract | Publisher Full Text
24. Van Dijk D, Manor O, Carey LB: Publication metrics and success on the academic job market. Curr. Biol. 2014; 24(11): 516–517. Publisher Full Text
25. Wang W, Yu S, Bekele TM, et al.: Scientific collaboration patterns vary with scholars’ academic ages. Scientometrics. 2017; 112(1): 329–343. Publisher Full Text
26. Sabharwal M: Comparing Research Productivity Across Disciplines and Career Stages. J. Comp. Policy Anal.: Res. Pract. 2013; 15(2): 141–163. Publisher Full Text
27. Sugimoto CR, Sugimoto TJ, Tsou A, et al.: Age stratification and cohort effects in scholarly communication: a study of social sciences. Scientometrics. 2016; 109(2): 997–1016. Publisher Full Text
28. Ioannidis JPA, Boyack KW, Klavans R: Estimates of the Continuously Publishing Core in the Scientific Workforce. PLoS One. 2014 Jul 9; 9(7): e101698. PubMed Abstract | Publisher Full Text
29. Calver M, Bryant K, Wardell-Johnson G: Quantifying the internationality and multidisciplinarity of authors and journals using ecological statistics. Scientometrics. 2018; 115(2): 731–748. Publisher Full Text
30. Nature: Editorial: Misspent youth. Nature. 2016; 538: 427.
31. De Meis L, Do Carmo MS, De Meis C: Impact factors: Just part of a research treadmill [2]. Nature. 2003; 424(6950): 723. PubMed Abstract | Publisher Full Text
32. Sills J: Research night owls. Science. 2016; 354(6315): 964. Publisher Full Text
33. Petersen AM, Riccaboni M, Stanley HE, et al.: Persistence and uncertainty in the academic career. Proc. Natl. Acad. Sci. U. S. A. 2012; 109(14): 5213–5218. PubMed Abstract | Publisher Full Text
34. Sarewitz D: The pressure to publish pushes down quality. Nature. 2016; 533: 147. PubMed Abstract | Publisher Full Text
35. Fanelli D, Larivière V: Researchers’ individual publication rate has not increased in a century. PLoS One. 2016; 11(3): 1–12. Publisher Full Text
36. Hosseini M: Equal Co-authorship Practices: Review and Recommendations. Sci. Eng. Ethics. 2020 Jun 1; 26(3): 1133–1148. PubMed Abstract | Publisher Full Text
37. Hsiehchen D, Espinoza M, Hsieh A: Multinational teams and diseconomies of scale in collaborative research. Sci. Adv. 2015; 1(8). PubMed Abstract | Publisher Full Text
38. Carpenter CR, Cone DC, Sarli CC: Using publication metrics to highlight academic productivity and research impact. Acad. Emerg. Med. 2014; 21(10): 1160–1172. PubMed Abstract | Publisher Full Text
39. Haslam N, Laham S: Early-career scientific achievement and patterns of authorship: The mixed blessings of publication leadership and collaboration. Res. Eval. 2009; 18(5): 405–410. Publisher Full Text
40. Shin JC, Cummings WK: Multilevel analysis of academic publishing across disciplines: Research preference, collaboration, and time on research. Scientometrics. 2010; 85(2): 581–594. Publisher Full Text
41. Zubieta AF: Recognition and weak ties: Is there a positive effect of postdoctoral position on academic performance and career development?. Res. Eval. 2009; 18(2): 105–115. Publisher Full Text
42. Scaffidi AK, Berman JE: A positive postdoctoral experience is related to quality supervision and career mentoring, collaborations, networking and a nurturing research environment. High. Educ. 2011; 62(6): 685–698. Publisher Full Text
43. Forret ML, Dougherty TW: Networking behaviors and career outcomes: Differences for men and women?. J. Organ. Behav. 2004; 25(3): 419–437. Publisher Full Text
44. Seibert SE, Kraimer ML, Liden RC: A Social Capital Theory of Career Success. Acad. Manag. J. 2001; 44(2): 219–237. Publisher Full Text
45. Kendal D: ECR publication metrics. Harvard Dataverse. 2021; V1: UNF:6:QyYUbjhtC0/cPgUAVetkcA== [fileUNF]. Publisher Full Text

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 26 Jan 2022

Author details Author details

¹ Healthy Landscapes Research Group, School of Geography, Planning and Spatial Sciences, University of Tasmania, Hobart, Tas, 7000, Australia
² School of Ecosystem and Forest Sciences, The University of Melbourne, Parkville, Vic, 3010, Australia
³ School of Life and Environmental Sciences, The University of Sydney, Sydney, NSW, 2006, Australia

Dave Kendal
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Kate E. Lee
Roles: Conceptualization, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing

Kylie Soanes
Roles: Conceptualization, Investigation, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Caragh G. Threlfall
Roles: Investigation, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

This work was supported by the Clean Air and Urban Landscapes hub (DK, KL, KS and CT) and the Threatened Species Recovery hub (KS), funded by the Australian Government’s National Environmental Science Program.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (1)

version 1

Published: 26 Jan 2022, 11:95

https://doi.org/10.12688/f1000research.75604.1

Copyright

© 2022 Kendal D et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

0

SEE MORE DETAILS

CITE

how to cite this article

Kendal D, Lee KE, Soanes K and Threlfall CG. ‘The great publication race’ vs ‘abandon paper counting’: Benchmarking ECR publication and co-authorship rates over past 50 years to inform research evaluation [version 1; peer review: 2 approved with reservations]. F1000Research 2022, 11:95 (https://doi.org/10.12688/f1000research.75604.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 26 Jan 2022

Views

17

Reviewer Report 14 Apr 2022

Laurel L Haak, Ronin Institute, Montclair, NJ, USA

Approved with Reservations

https://doi.org/10.5256/f1000research.79492.r134547

This study examines options for measuring outputs of ECRs, to get away from over-reliance on paper counting in researcher evaluation. It uses publication data and normalizes mean and median publication volumes based on active years.

The paper ... Continue reading

This study examines options for measuring outputs of ECRs, to get away from over-reliance on paper counting in researcher evaluation. It uses publication data and normalizes mean and median publication volumes based on active years.

The paper could be further improved by addressing the following points:

While the paper presents well-founded proposals for assessing ECR output, it still relies fully on publication data.
The authors do not address possible confounds with author disambiguation issues in the methods or results section.
The data made available do not appear to include R scripts or author data.
The introduction and conclusion sections could have referenced more of the recent literature on team science and collaborative research trends.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Research workforce policy, research infrastructure, collaboration, team science, author disambiguation, research evaluation

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Views

20

Reviewer Report 07 Mar 2022

Gerben ter Riet, Urban Vitality Centre of Expertise, Amsterdam University of Applied Sciences, Amsterdam, The Netherlands; Cardiology Department, Amsterdam University Medical Center, Amsterdam, The Netherlands

Approved with Reservations

https://doi.org/10.5256/f1000research.79492.r124516

The authors set out to provide the debate(s) on academic (over)production, slow(er) science and better work-life balance with empirical evidence, focusing on early career researchers (ECRs). In particular, they produced reference values (benchmarks) for what may be the expected number ... Continue reading

The authors set out to provide the debate(s) on academic (over)production, slow(er) science and better work-life balance with empirical evidence, focusing on early career researchers (ECRs). In particular, they produced reference values (benchmarks) for what may be the expected number of papers an average ECR publishes, stratified by his/her year after career start. These benchmarks, according to the authors, may serve in research evaluations of ECRs.

In addition, they show how these production counts depend on the number and variability of co-authors on papers. Finally, they show by how much the number of publications ECRs produce has changed (increased) over the last 5 decades.

To me paper would gain much focus if it dealt with objective 2 only:

Developing evidence-based benchmarks for ECR publication rates as careers develop.

That is a rich topic and worthwhile goal in itself.

My personal opinion is that: I sympathize with the idea that science should slow down and focus on quality more. In the same vein, I worry that the benchmarks the authors produce, if valid, might also be used against ECRs by employers or funders that require an ECR to be at least above the 75th quantile. So, a more radical approach in research evaluations is that taken by the Dutch scientific authorities where they do away with counts altogether. (I am not a co-author of these guidelines). https://www.universiteitenvannederland.nl/files/documenten/Domeinen/Onderzoek/SEP_2021-2027.pdf (in English).

In their balanced discussion, the authors acknowledge their work as a first step and also emphasize that: "Further research could help to better understand which disciplines these benchmarks are indeed valid for, and which disciplines require alternative benchmarks (or for which publishing benchmarks may not be appropriate)."

Patrick Royston (1995)¹ has developed a method to derive benchmarks (reference values) in which covariables can be incorporated. I wonder if the authors might benefit from Royston’s methodology and derive model-based benchmarks including 95% confidence intervals around the best fitting line as well as around the lines that form the x-th (e.g. 90th) centile ranges.

Below I follow the various paragraphs in the paper:

Introduction, paragraph 1

What is exactly meant by “managing academic expectations”. Reference 1 does not mention this term as far as I could see. This should be clarified.
The authors switch from a ‘focus’ to an ‘unbalanced focus’ on publication counts, but I think that it is important that the authors distinguish these, because some (limited) usage of metrics is quite different – and may be defensible – from evaluations in which they are central.
The authors should state far more clearly what they see as quality and impact (or valid indicators of those)? I think that publication counts is a far more primitive measure than citations (not self-citations!) and journal impact factor and should not be in the same category. The author seem to acknowledge this in paragraph 2 where they use citation rates as a measure of quality that is not all bad, saying: “It is a poor predictor of later research ‘quality’ as measured by indicators such as citation rates. 19 ”, but its condemnation here and approval in paragraph 2, to me, is inconsistent.
The authors worry that evaluations may be reduced to competitions of researcher status. But to me, researcher status is something inherent to any evaluation system? Whatever systems for evaluation we build, each will produce its champions. In ideal (?) circumstances, we may view evaluations as mere starting points for a discussion with others (your boss, funders, ..) on what is needed for ECRs or institutes to become (even) better, not as justifications to stop funding or firing someone. But is that realistic?

Introduction, paragraph 2

I wonder if the h-index is also a bad measure if it were used to compare researchers within, say, a 3-year working career range, a career-age stratified h-index? If this cannot be realistically done, the authors should argue along these lines.
Publishing with top scientists can be an asset of an ECR, but will usually occur via the network of one of his/her seniors, which may be an indicator of “working at a more prestigious institution”. But, anyhow, having access to such people may be an, albeit, indirect indicator of an ECR’s own quality, since prestigious institutions may apply heavier hurdles to become a member.
The authors should distinguish between the measurement of an ECR’s success today and previously measured predictors (future) of having success. A combination of both may convey additional information, since e.g. having no success today while having had a favorable prediction score in the past may indicate that despite an optimal academic environment one did not succeed. The reverse of course is an excellent ECR who has been working in a modest environment and is nevertheless successful today, which may highlight exceptional qualities (or luck).
“In the context of uncapped publication expectations”. I think that the authors should distinguish between personal evaluations within research institutes or when seeking funds and (group or team) evaluations of institutes of which an ECR may be a member. In The Netherlands, the SEP intended for use between 2021 and 2027 for research institutes de-emphasizes count outcomes and h-indices (seemingly ignoring career duration stratified h-indices) and has moved toward adopting group narratives, although it still includes research grants awarded to individuals, grants awarded to major collaborative research projects and prizes awarded to individuals or collaborative research projects: https://www.universiteitenvannederland.nl/files/documenten/Domeinen/Onderzoek/SEP_2021-2027.pdf (in English). I struggled to grasp what the customs are in Australia, and concluded that institutes have the freedom to take a sample of their output and present for formal evaluation.

Introduction, paragraph 3

The authors write “There is currently little empirical data to make sense of these debates for ECRs..” I think that some of the confusion (or lack of focus) in the introduction stems from not being explicit on which debate the authors want to focus on exactly. It seems they want to derive reference (‘normal) ’values (“career stage-specific benchmarks”, see study objectives) for publication counts (like labs have reference values for blood constituents), but how does this match with their critique of such counts? The authors should clarify their stance on this.

Introduction, paragraph 4

The authors state that “A research evaluation framework that is specific to the early years of a career is needed. This should be based on an understanding of how ECR publication and co-authorship rates have changed through time, how they progress during a career, and what strategies best support research ‘success’.” Let’s suppose that such a framework is indeed needed, why should it be based on understanding publication rates through time or how they progress over careers? The authors should provide compelling arguments for their assertions.
I am not convinced that their research questions 3 make sense. Can the authors explain why estimating the effect of such strategies is better than incorporating the strategies into the benchmarks themselves which to me seems a more logical approach?
Given that citation rates and journal impact factors may be somewhat more refined measures than pure publication counts, wouldn’t it be more valuable to develop benchmarks for the former two indicators? Please comment.
How were trends, if any, in part-time working factors accounted for? The authors mention this as a limitation in their discussion on the benchmarks, but do not in their discussion of the trends over time.
The study may suffer from selection bias (survivorship bias to be specific) (and the authors acknowledge this, as a limitation, although not using this term): first, ECRs who dropped out because they couldn't accept the current environment of hyper-competition and authors who did not publish anything in 2019 were not included in the study sample.
Related to the point above, the authors state in their Discussion that: "Concerningly, the real size of the observed acceleration in publication rates is likely to be greater than described here. Our data is likely to be oversampling the most successful established scientists, as ECR’s from earlier decades who ceased publishing before 2019 are not included." I drew the exact opposite conclusion and think that the observed (real!) acceleration is smaller than described here, because those who did (and do) not survive in the current hypercompetitive environment have been left out. The authors should clarify their line of thought.
How might the authors account for delays in publishing between journals or fields (humanities, math)? If not, they should add this as a limitation.
Did the authors cut out time gaps with zero publication using calendar years or years since last date of publication and did they use dates of submission or dates of publication? Please clarify.
Were retractions taken into account or are these numbers negligible.

Results section.

In figure 5, the 95% confidence intervals are missing.

Data and syntaxes.

Please add the code to extract the data from Scopus and the exact R code used for extractions.

Is the work clearly and accurately presented and does it cite the current literature?

No
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

References

1. Royston P: Calculation of unconditional and conditional reference intervals for foetal size and growth from longitudinal measurements.Stat Med. 1995; 14 (13): 1417-36 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Clinical epidemiology, meta-research, research integrity, publication bias

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 26 Jan 2022

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 1 26 Jan 22	read	read

Gerben ter Riet, Amsterdam University of Applied Sciences, Amsterdam, The Netherlands; Amsterdam University Medical Center, Amsterdam, The Netherlands
Laurel L Haak, Ronin Institute, Montclair, USA

Comments on this article

All Comments(0)

Add a comment

Sign up for content alerts

Browse by related subjects

Back to all reports

Reviewer Report

17 Views

14 Apr 2022 | for Version 1

Laurel L Haak, Ronin Institute, Montclair, NJ, USA

17 Views Cite this report Responses(0)

Approved With Reservations

This study examines options for measuring outputs of ECRs, to get away from over-reliance on paper counting in researcher evaluation. It uses publication data and normalizes mean and median publication volumes based on active years.

The paper could be further improved by addressing the following points:

While the paper presents well-founded proposals for assessing ECR output, it still relies fully on publication data.
The authors do not address possible confounds with author disambiguation issues in the methods or results section.
The data made available do not appear to include R scripts or author data.
The introduction and conclusion sections could have referenced more of the recent literature on team science and collaborative research trends.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Research workforce policy, research infrastructure, collaboration, team science, author disambiguation, research evaluation

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

20 Views

07 Mar 2022 | for Version 1

Gerben ter Riet, Urban Vitality Centre of Expertise, Amsterdam University of Applied Sciences, Amsterdam, The Netherlands; Cardiology Department, Amsterdam University Medical Center, Amsterdam, The Netherlands

20 Views Cite this report Responses(0)

Approved With Reservations

The authors set out to provide the debate(s) on academic (over)production, slow(er) science and better work-life balance with empirical evidence, focusing on early career researchers (ECRs). In particular, they produced reference values (benchmarks) for what may be the expected number of papers an average ECR publishes, stratified by his/her year after career start. These benchmarks, according to the authors, may serve in research evaluations of ECRs.

In addition, they show how these production counts depend on the number and variability of co-authors on papers. Finally, they show by how much the number of publications ECRs produce has changed (increased) over the last 5 decades.

To me paper would gain much focus if it dealt with objective 2 only:

Developing evidence-based benchmarks for ECR publication rates as careers develop.

That is a rich topic and worthwhile goal in itself.

My personal opinion is that: I sympathize with the idea that science should slow down and focus on quality more. In the same vein, I worry that the benchmarks the authors produce, if valid, might also be used against ECRs by employers or funders that require an ECR to be at least above the 75th quantile. So, a more radical approach in research evaluations is that taken by the Dutch scientific authorities where they do away with counts altogether. (I am not a co-author of these guidelines). https://www.universiteitenvannederland.nl/files/documenten/Domeinen/Onderzoek/SEP_2021-2027.pdf (in English).

In their balanced discussion, the authors acknowledge their work as a first step and also emphasize that: "Further research could help to better understand which disciplines these benchmarks are indeed valid for, and which disciplines require alternative benchmarks (or for which publishing benchmarks may not be appropriate)."

Patrick Royston (1995)¹ has developed a method to derive benchmarks (reference values) in which covariables can be incorporated. I wonder if the authors might benefit from Royston’s methodology and derive model-based benchmarks including 95% confidence intervals around the best fitting line as well as around the lines that form the x-th (e.g. 90th) centile ranges.

Below I follow the various paragraphs in the paper:

Introduction, paragraph 1

What is exactly meant by “managing academic expectations”. Reference 1 does not mention this term as far as I could see. This should be clarified.
The authors switch from a ‘focus’ to an ‘unbalanced focus’ on publication counts, but I think that it is important that the authors distinguish these, because some (limited) usage of metrics is quite different – and may be defensible – from evaluations in which they are central.
The authors should state far more clearly what they see as quality and impact (or valid indicators of those)? I think that publication counts is a far more primitive measure than citations (not self-citations!) and journal impact factor and should not be in the same category. The author seem to acknowledge this in paragraph 2 where they use citation rates as a measure of quality that is not all bad, saying: “It is a poor predictor of later research ‘quality’ as measured by indicators such as citation rates. 19 ”, but its condemnation here and approval in paragraph 2, to me, is inconsistent.
The authors worry that evaluations may be reduced to competitions of researcher status. But to me, researcher status is something inherent to any evaluation system? Whatever systems for evaluation we build, each will produce its champions. In ideal (?) circumstances, we may view evaluations as mere starting points for a discussion with others (your boss, funders, ..) on what is needed for ECRs or institutes to become (even) better, not as justifications to stop funding or firing someone. But is that realistic?

Introduction, paragraph 2

I wonder if the h-index is also a bad measure if it were used to compare researchers within, say, a 3-year working career range, a career-age stratified h-index? If this cannot be realistically done, the authors should argue along these lines.
Publishing with top scientists can be an asset of an ECR, but will usually occur via the network of one of his/her seniors, which may be an indicator of “working at a more prestigious institution”. But, anyhow, having access to such people may be an, albeit, indirect indicator of an ECR’s own quality, since prestigious institutions may apply heavier hurdles to become a member.
The authors should distinguish between the measurement of an ECR’s success today and previously measured predictors (future) of having success. A combination of both may convey additional information, since e.g. having no success today while having had a favorable prediction score in the past may indicate that despite an optimal academic environment one did not succeed. The reverse of course is an excellent ECR who has been working in a modest environment and is nevertheless successful today, which may highlight exceptional qualities (or luck).
“In the context of uncapped publication expectations”. I think that the authors should distinguish between personal evaluations within research institutes or when seeking funds and (group or team) evaluations of institutes of which an ECR may be a member. In The Netherlands, the SEP intended for use between 2021 and 2027 for research institutes de-emphasizes count outcomes and h-indices (seemingly ignoring career duration stratified h-indices) and has moved toward adopting group narratives, although it still includes research grants awarded to individuals, grants awarded to major collaborative research projects and prizes awarded to individuals or collaborative research projects: https://www.universiteitenvannederland.nl/files/documenten/Domeinen/Onderzoek/SEP_2021-2027.pdf (in English). I struggled to grasp what the customs are in Australia, and concluded that institutes have the freedom to take a sample of their output and present for formal evaluation.

Introduction, paragraph 3

The authors write “There is currently little empirical data to make sense of these debates for ECRs..” I think that some of the confusion (or lack of focus) in the introduction stems from not being explicit on which debate the authors want to focus on exactly. It seems they want to derive reference (‘normal) ’values (“career stage-specific benchmarks”, see study objectives) for publication counts (like labs have reference values for blood constituents), but how does this match with their critique of such counts? The authors should clarify their stance on this.

Introduction, paragraph 4

The authors state that “A research evaluation framework that is specific to the early years of a career is needed. This should be based on an understanding of how ECR publication and co-authorship rates have changed through time, how they progress during a career, and what strategies best support research ‘success’.” Let’s suppose that such a framework is indeed needed, why should it be based on understanding publication rates through time or how they progress over careers? The authors should provide compelling arguments for their assertions.
I am not convinced that their research questions 3 make sense. Can the authors explain why estimating the effect of such strategies is better than incorporating the strategies into the benchmarks themselves which to me seems a more logical approach?
Given that citation rates and journal impact factors may be somewhat more refined measures than pure publication counts, wouldn’t it be more valuable to develop benchmarks for the former two indicators? Please comment.
How were trends, if any, in part-time working factors accounted for? The authors mention this as a limitation in their discussion on the benchmarks, but do not in their discussion of the trends over time.
The study may suffer from selection bias (survivorship bias to be specific) (and the authors acknowledge this, as a limitation, although not using this term): first, ECRs who dropped out because they couldn't accept the current environment of hyper-competition and authors who did not publish anything in 2019 were not included in the study sample.
Related to the point above, the authors state in their Discussion that: "Concerningly, the real size of the observed acceleration in publication rates is likely to be greater than described here. Our data is likely to be oversampling the most successful established scientists, as ECR’s from earlier decades who ceased publishing before 2019 are not included." I drew the exact opposite conclusion and think that the observed (real!) acceleration is smaller than described here, because those who did (and do) not survive in the current hypercompetitive environment have been left out. The authors should clarify their line of thought.
How might the authors account for delays in publishing between journals or fields (humanities, math)? If not, they should add this as a limitation.
Did the authors cut out time gaps with zero publication using calendar years or years since last date of publication and did they use dates of submission or dates of publication? Please clarify.
Were retractions taken into account or are these numbers negligible.

Results section.

In figure 5, the 95% confidence intervals are missing.

Data and syntaxes.

Please add the code to extract the data from Scopus and the exact R code used for extractions.

Is the work clearly and accurately presented and does it cite the current literature?

No
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

References

1. Royston P: Calculation of unconditional and conditional reference intervals for foetal size and growth from longitudinal measurements.Stat Med. 1995; 14 (13): 1417-36 PubMed Abstract | Publisher Full Text

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Clinical epidemiology, meta-research, research integrity, publication bias

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Respond to this report

Responses (0)

[1] 1. Chapman CA, Bicca-Marques JC, Calvignac-Spencer S, et al.: Games academics play and their consequences: How authorship, h-index and journal impact factors are shaping the future of academia. Proc. R. Soc. B Biol. Sci. 2019; 286(1916). Publisher Full Text

[2] 2. Hicks D, Wouters P, Waltman L, et al.: The Leiden Manifesto for research metrics. Use these ten principles to guide research evaluation. Nature. 2015; 520(7548): 9–11. Publisher Full Text

[3] 3. Finkel A: The road to bad research is paved with good intentions. Nature. 2019; 566(7744): 297. PubMed Abstract | Publisher Full Text

[4] 4. Abbott A, Cyranoski D, Jones N, et al.: Metrics: Do metrics matter?. Nature. 2010; 465(7300): 860–862. Publisher Full Text

[5] 5. Way SF, Morgan AC, Clauset A, et al.: The misleading narrative of the canonical faculty productivity trajectory. Proc. Natl. Acad. Sci. U. S. A. 2017; 114(44): E9216–E9223. PubMed Abstract | Publisher Full Text

[6] 6. Tregoning J: How will you judge me if not by impact factor?. Nature. 2018; 558: 345. PubMed Abstract | Publisher Full Text

[7] 7. Schneider JW, Aagaard K, Bloch CW: What happens when national research funding is linked to differentiated publication counts? A comparison of the Australian and Norwegian publication-based funding models. Res. Eval. 2016; 25(3): 244–256. Publisher Full Text

[8] 8. Butler L: Modifying publication practices in response to funding formulas. Res. Eval. 2003; 12(1): 39–46. Publisher Full Text

[9] 9. Paasche Ø, Österblom H: Unsustainable Science. One Earth. 2019; 1(1): 39–42. Publisher Full Text

[10] 10. Frith U: Fast Lane to Slow Science. Trends Cogn. Sci. 2020; 24(1): 1–2. PubMed Abstract | Publisher Full Text

[11] 11. Carbon CC: The carbon_h-factor: Predicting individuals’ research impact at early stages of their career. PLoS One. 2011; 6(12). PubMed Abstract | Publisher Full Text

[12] 12. Gruber T: Academic sell-out: how an obsession with metrics and rankings is damaging academia. J. Mark. High. Educ. 2014; 24(2): 165–177. Publisher Full Text

[13] 13. Ayaz S, Masood N, Islam MA: Predicting scientific impact based on h-index. Scientometrics. 2018; 114(3): 993–1010. Publisher Full Text

[14] 14. Laurance WF, Useche DC, Laurance SG, et al.: Predicting Publication Success for Biologists. Bioscience. 2013; 63(10): 817–823. Publisher Full Text

[15] 15. Zeng A, Shen Z, Zhou J, et al.: The science of science: From the perspective of complex systems. Phys. Rep. 2017; 714–715: 1–73. Publisher Full Text

[16] 16. Zhang Y, Yu Q: What is the best article publishing strategy for early career scientists?. Scientometrics. 2020; 122(1): 397–408. Publisher Full Text

[17] 17. Li W, Aste T, Caccioli F, et al.: Early coauthorship with top scientists predicts success in academic careers. Nat. Commun. 2019; 10(1): 1–9. Publisher Full Text

[18] 18. Powell K: Young, talented and fed-up: Scientists tell their stories. Nature. 2016; 538(7626): 446–449. PubMed Abstract | Publisher Full Text

[19] 19. Bornmann L, Williams R: Can the journal impact factor be used as a criterion for the selection of junior researchers? A large-scale empirical study based on ResearcherID data. J. Informet. 2017; 11(3): 788–799. Publisher Full Text

[20] 20. Shen H: Plight of the postdoc. Nature. 2015; 525: 279–281. PubMed Abstract | Publisher Full Text

[21] 21. Small GE: The Post-doc dilemma. Nature. 2012; 483: 235. Publisher Full Text

[22] 22. Waaijer CJF, Macaluso B, Sugimoto CR, et al.: Stability and longevity in the publication careers of U.S. doctorate recipients. PLoS One. 2016; 11(4): 1–15. Publisher Full Text

[23] 23. Petersen AM, Jung WS, Yang JS, et al.: Quantitative and empirical demonstration of the Matthew effect in a study of career longevity. Proc. Natl. Acad. Sci. U. S. A. 2011; 108(1): 18–23. PubMed Abstract | Publisher Full Text

[24] 24. Van Dijk D, Manor O, Carey LB: Publication metrics and success on the academic job market. Curr. Biol. 2014; 24(11): 516–517. Publisher Full Text

[25] 25. Wang W, Yu S, Bekele TM, et al.: Scientific collaboration patterns vary with scholars’ academic ages. Scientometrics. 2017; 112(1): 329–343. Publisher Full Text

[26] 26. Sabharwal M: Comparing Research Productivity Across Disciplines and Career Stages. J. Comp. Policy Anal.: Res. Pract. 2013; 15(2): 141–163. Publisher Full Text

[27] 27. Sugimoto CR, Sugimoto TJ, Tsou A, et al.: Age stratification and cohort effects in scholarly communication: a study of social sciences. Scientometrics. 2016; 109(2): 997–1016. Publisher Full Text

[28] 28. Ioannidis JPA, Boyack KW, Klavans R: Estimates of the Continuously Publishing Core in the Scientific Workforce. PLoS One. 2014 Jul 9; 9(7): e101698. PubMed Abstract | Publisher Full Text

[29] 29. Calver M, Bryant K, Wardell-Johnson G: Quantifying the internationality and multidisciplinarity of authors and journals using ecological statistics. Scientometrics. 2018; 115(2): 731–748. Publisher Full Text

[30] 30. Nature: Editorial: Misspent youth. Nature. 2016; 538: 427.

[31] 31. De Meis L, Do Carmo MS, De Meis C: Impact factors: Just part of a research treadmill [2]. Nature. 2003; 424(6950): 723. PubMed Abstract | Publisher Full Text

[32] 32. Sills J: Research night owls. Science. 2016; 354(6315): 964. Publisher Full Text

[33] 33. Petersen AM, Riccaboni M, Stanley HE, et al.: Persistence and uncertainty in the academic career. Proc. Natl. Acad. Sci. U. S. A. 2012; 109(14): 5213–5218. PubMed Abstract | Publisher Full Text

[34] 34. Sarewitz D: The pressure to publish pushes down quality. Nature. 2016; 533: 147. PubMed Abstract | Publisher Full Text

[35] 35. Fanelli D, Larivière V: Researchers’ individual publication rate has not increased in a century. PLoS One. 2016; 11(3): 1–12. Publisher Full Text

[36] 36. Hosseini M: Equal Co-authorship Practices: Review and Recommendations. Sci. Eng. Ethics. 2020 Jun 1; 26(3): 1133–1148. PubMed Abstract | Publisher Full Text

[37] 37. Hsiehchen D, Espinoza M, Hsieh A: Multinational teams and diseconomies of scale in collaborative research. Sci. Adv. 2015; 1(8). PubMed Abstract | Publisher Full Text

[38] 38. Carpenter CR, Cone DC, Sarli CC: Using publication metrics to highlight academic productivity and research impact. Acad. Emerg. Med. 2014; 21(10): 1160–1172. PubMed Abstract | Publisher Full Text

[39] 39. Haslam N, Laham S: Early-career scientific achievement and patterns of authorship: The mixed blessings of publication leadership and collaboration. Res. Eval. 2009; 18(5): 405–410. Publisher Full Text

[40] 40. Shin JC, Cummings WK: Multilevel analysis of academic publishing across disciplines: Research preference, collaboration, and time on research. Scientometrics. 2010; 85(2): 581–594. Publisher Full Text

[41] 41. Zubieta AF: Recognition and weak ties: Is there a positive effect of postdoctoral position on academic performance and career development?. Res. Eval. 2009; 18(2): 105–115. Publisher Full Text

[42] 42. Scaffidi AK, Berman JE: A positive postdoctoral experience is related to quality supervision and career mentoring, collaborations, networking and a nurturing research environment. High. Educ. 2011; 62(6): 685–698. Publisher Full Text

[43] 43. Forret ML, Dougherty TW: Networking behaviors and career outcomes: Differences for men and women?. J. Organ. Behav. 2004; 25(3): 419–437. Publisher Full Text

[44] 44. Seibert SE, Kraimer ML, Liden RC: A Social Capital Theory of Career Success. Acad. Manag. J. 2001; 44(2): 219–237. Publisher Full Text

[45] 45. Kendal D: ECR publication metrics. Harvard Dataverse. 2021; V1: UNF:6:QyYUbjhtC0/cPgUAVetkcA== [fileUNF]. Publisher Full Text

‘The great publication race’ vs ‘abandon paper counting’: Benchmarking ECR publication and co-authorship rates over past 50 years to inform research evaluation

Abstract

Keywords

Introduction

Methods

Author samples

Table 1. Journals used in the disciplinary author sample.

Data collection

Analyses

Results

Are ECR publication rates changing over time?

Figure 1. Prediction of publications per year (A and C) and first author publications (B and D) by first year of publication for our multidisciplinary author sample (A and B) and our disciplinary sample (C and D).

Figure 2. Prediction of average co-authors per year (A and C) and author dissimilarity (B and D) by first year of publication for our multidisciplinary author sample (A and B) and our disciplinary sample (C and D).

Figure 3. A – Mean publications per year by active year (solid circle) and career age (open square) for all authors. B – The proportion (bars) and cumulative proportion (circles) of authors with no publications in some years over the first 10 active years of their career.

Evidence-based benchmarks for ECR publication rates

Table 2. Mean and median publications rates (publications/year) at different career ages for 55,332 authors who published in Nature, Science and PNAS in 2019.

Figure 4. A boxplot of publications per year by active years of publishing.

The effect of co-authorship strategies on the probability of researchers meeting these benchmarks

Table 3. Model fit statistics.

Discussion

Changes in ECR publication rates over time

Evidence-based benchmarks for ECR publication rates

The effect of co-authorship strategies on the probability of researchers meeting these benchmarks

Study limitations

Conclusion

Data availability

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated