ALL Metrics
-
Views
-
Downloads
Get PDF
Get XML
Cite
Export
Track
Research Note

F1000Prime: an analysis of discipline-specific reader data from Mendeley

[version 1; peer review: 1 approved with reservations, 1 not approved]
PUBLISHED 11 Feb 2015
Author details Author details
OPEN PEER REVIEW
REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

Abstract

We have used the F1000Prime recommended paper set (n= 114,582 biomedical papers) to inquire the number of Mendeley readers per (sub-) discipline via the Mendeley Application Programming Interface (API). Although the (sub-) discipline of Mendeley readers is self-assigned and not mandatory, we find that a large share (99.9%) of readers at Mendeley does share their (sub-) discipline. As expected, we find most readers of F1000Prime recommended papers work in the disciplines of biology and medicine. A network analysis reveals strong connections between the disciplines of engineering, chemistry, physics, biology, and medicine.

Keywords

F1000Prime, Altmetrics, Mendeley, paper, evaluation

Introduction

Interest in the broad impact of research (Bornmann, 2012, 2013) has resulted in new forms of impact measurements. Traditional forms of impact measurements using bibliometrics only allow the measurement of impact on research itself. These new forms which have been named as altmetrics (abbreviation of alternative metrics) pretend to measure the impact of research on other areas of society (than research) by counting the mentions of papers in social media: “Alternative metrics, sometimes shortened to just altmetrics, is an umbrella term covering new ways of approaching, measuring and providing evidence for impact” (Adie, 2014, p. 349). As altmetrics, the number of readers (on Mendeley), mirco-bloggers (on Twitter), and other consumers of research using social media are counted. Although scientometrics research on altmetrics is still in a very early phase (comparable to research on bibliometrics in the 1970s), the use of these data in research evaluation is already an issue. For example, altmetrics is considered in the Snowball Metrics project (Colledge, 2014). This project compiled a set of clearly defined indicators which will be used by participating universities (mostly Anglo-American universities) for research evaluation purposes. It seems that altmetrics will be used in practice before scientometrics research has produced standards on their reliable, fair and valid application (Weller, 2015).

This study uses one of the most important sources for altmetrics data, namely Mendeley. Mendeley “claims 3.1 million members. It was originally launched as software for managing and storing documents, but it encourages private and public social networking” (Van Noorden, 2014, p. 126). Since data from Mendeley can be received by an Application Programming Interface (API) without any problems and the coverage of the scientific literature has been pointed out as high (Priem, 2014), Mendeley is a very attractive data source for the reception of research. “Mendeley records the number of users that have listed it [i.e. an article], describing them as readers, whether or not they actually read it. Presumably, listing an article in Mendeley tends to reflect that an article has been read or will be read in the future, although there is no evidence that this assumption is true” (Thelwall & Maflahi, in press).

In this study, we match Mendeley data with data from F1000Prime. F1000Prime is a database with biomedical papers and their reviews by peers. It is intended as a support tool for researchers to receive hints for the most important literature. Since it is not clear who actually reads the F1000Prime recommended papers, we investigated the disciplines of researchers (and other people) who have read these papers. We are mainly interested in two questions: are F1000Prime papers only read by people from biomedicine or are people from other disciplines also interested? Which disciplines read F1000Prime papers frequently or seldom together? The latter question will be answered by using social network techniques.

Methods

Peer ratings provided by F1000Prime

F1000Prime is a post-publication peer review system of papers from medical and biological journals. This service is part of the Science Navigation Group, which publishes and develops information services for the professional biomedical community and the consumer market. Papers for F1000Prime are selected by a peer-nominated global "Faculty" of leading scientists and clinicians. The Faculty members rate the papers and explain their importance. This means that only a selected set of papers from the biomedical area covered is reviewed, and most of the papers are actually not (Kreiman & Maunsell, 2011; Wouters & Costas, 2012).

The Faculty nowadays numbers more than 5,000 members worldwide, assisted by further associates, which are organised into more than 40 subjects. Members can choose and evaluate any paper of interest; however, "the great majority pick papers published within the past month, including advance online papers, meaning that users can be made aware of important papers rapidly" (Wets et al., 2003, p. 254). Although many papers published in popular and high-profile journals (e.g. Nature, New England Journal of Medicine, Science) are evaluated, 85% of the papers selected come from specialised or less well-known journals (Wouters & Costas, 2012). The F1000Prime database is regarded as a useful aid for researchers (and other people working research-oriented) to obtain indications of the most relevant papers in the biomedical area: "The aim of Faculty of 1000 is not to provide an evaluation for all papers, as this would simply exacerbate the ‘noise’, but to take advantage of electronic developments to create the optimal human filter for effectively reducing the noise" (Wets et al., 2003, p. 253).

Use of the Mendeley API

Within the first half of 2014 the reference manager Mendeley provided a new version of its API. Some restrictions of the previous API were lifted. For example, the usage statistics were previously provided in relative terms and only for the top three entries (Haustein & Larivière, 2014). The new API provides results in absolute numbers and not only for the top three but for all entries. Mendeley provides access to the readership status (e. g. professor, postdoc, or student) and the distribution of the Mendeley readership across scientific disciplines as well as countries via the API. Those sets of data can be correlated with other information available about papers (e. g. citations or Twitter counts).

Before one can start to use the Mendeley API, one has to register as a Mendeley user. Afterwards, registration of the desired application is necessary (http://dev.mendeley.com). Authentication with the API is done via OAuth 2.0. The credentials are set during registration of the application.

We used R (http://www.r-project.org/) to interface to the Mendeley API. It seems to us that using other interfaces does not change the functionality or responsiveness, but we did not try to use other interfaces. Mendeley provides sample codes for Javascript, Python, R, and Ruby (http://dev.mendeley.com/code/sample_code.html), whereby all requests to the API use HTTP GET and POST requests. Therefore, we suppose that any other scripting or programming language may be used. The reply is sent in Javascript Object Notation (JSON).

We requested user statistics for the F1000Prime publication set (n = 114,582 papers) using the PubMedID and DOI between the 4th and 6th of December 2014. We observed seemingly random connection problems. Sometimes those problems occurred after a few hundred or a few thousand requests. The largest chunk of requests we were able to get through the API without connection problems consisted of 47,629 papers. This large number of records is contrasted with smaller chunks of requests (between 1,049 and 9,307 records).

Mendeley provides a breakdown of the user count into sub-disciplines. The possible values for disciplines and sub-disciplines can be obtained directly from the API via the GET /disciplines endpoint. Each discipline has a certain number of sub-disciplines. The sub-discipline “miscellaneous” occurs in every discipline. Each Mendeley user can select a discipline and a sub-discipline from a drop-down menu. This piece of information is not mandatory, like the user’s location.

Network analysis

Pajek is used to create the F1000Prime readership network (http://pajek.imfm.si/doku.php; de Nooy, Mrvar & Batagelj, 2011) applying the spring embedder of Kamada & Kawai (1989). For detecting communities in the common readership of F1000Prime recommended papers, we used the VOS Clustering algorithm (Waltman, van Eck & Noyons, 2010), which is available in Pajek. The aim of this algorithm is to provide further insights into the structure of the network (Milojević, 2014).

Results

We found 6,263,913 Mendeley readers for the F1000Prime publication set. 99.9% (n=6,257,603) of them share their discipline and sub-discipline. This is a much higher percentage than those who share their geographical location (Haunschild, Stefaner & Bornmann, in preparation). For the F1000Prime publication set, the vast majority (74.94%) of Mendeley users is found in the “miscellaneous” sub-discipline of all disciplines. Therefore, we added up all the readers of all sub-disciplines for each discipline. The results of our study are presented in Table 1. Nine disciplines have at least 1% of the readers of the F1000Prime publication set. The remaining 16 disciplines have less than 1%. As expected, most readers (81.78%) of the F1000Prime literature assign themselves to the biomedical (sub-) disciplines. All other disciplines comprise the remaining 15.19% of the F1000Prime readership at Mendeley. The third largest readership is found in the discipline psychology which is related to medicine. After chemistry, which is also related to biology, five other disciplines show readership values above 1% within the F1000Prime literature. Those disciplines seem rather unrelated to the field of biomedical research, especially environmental sciences (according to Figure 1, see also the description below). 3.05% of the F1000Prime readers at Mendeley come from other disciplines (not shown in Table 1). The disciplines with most readers below the threshold of 1% are: social sciences (0.67%), mathematics (0.42%), electrical and electronic engineering (0.27%), education (0.22%), and materials sciences (0.21%).

Table 1. Mendeley users from different disciplines reading F1000Prime recommended papers.

Only disciplines are shown with more than 1% readers (sorted in decreasing order).

DisciplineMendeley readers% of Mendeley readers
of F1000Prime recommended papers
Biology4,149,85766.32
Medicine967,41215.46
Psychology248,7463.98
Chemistry182,7732.92
Environmental sciences169,2872.71
Engineering100,6361.61
Computer and information science96,4031.54
Physics88,5861.42
Earth sciences62,9841.01
Others190,9193.05
2ac1952f-5b27-4a20-af97-07a9fa224558_figure1.gif

Figure 1.

Network of F1000Prime recommended readers from arts and literature (AnL), astronomy and astrophysics (AsAs), biology (Bio), business administration (BuAd), chemistry (Chem), computer and information science (CIS), design (Des), earth sciences (ESci), economics (Eco), education (Edu), electrical and electronic engineering (EEE), engineering (Eng), environmental sciences (Env), humanities (Hum), law (Law), linguistics (Ling), management (Man), materials sciences (Mate), mathematics (Math), medicine (Med), philosophy (Phil), physics (Phys), psychology (Psy), social sciences (SoSc), sports and recreation (SpRe).

We also analyzed connections between the disciplines. These are shown in Figure 1. A paper which is read by Mendeley users of different disciplines (e.g. biology and physics) constitutes a connection between these disciplines. Therefore, a paper which is read by Mendeley users of the same discipline does not contribute to the network system, but a paper which is read by Mendeley users of many different disciplines contributes many connections to the network. The size of the vertices in Figure 1 reflects the numbers of readers for each discipline. The thicker and darker the edges between two disciplines, the more frequently they have read a F1000Prime paper jointly. The location of the discipline vertex also informs about the connectivity. The closer the vertex is located towards the center, the more connections to different disciplines are found. There are 25 disciplines and 300 links among those disciplines in the dataset. With a density of 1, the network is rather dense. The average node degree is 24. According to Figure 1, the strongest connection shows up between biology (Bio) and medicine (Med). The disciplines computers and information science (CIS), engineering (Eng), and chemistry (Chem) have rather strong connections to biology (Bio) and medicine (Med). The discipline arts and literature (AnL) shows a low amount of readers (0.15%, close to sports and recreation with 0.16%) as well as a good connection to other disciplines in the network.

The community detection algorithm detected two communities in the network with biology, medicine, engineering, chemistry, and physics as one community (yellow vertices) and all other disciplines as the other (green vertices).

Discussion

The (sub-) discipline of Mendeley readers is self-assigned and not mandatory. Still, we found that a large share (99.9%) of F1000Prime paper readers at Mendeley share their (sub-) discipline. Most readers (74.94%) assign the “miscellaneous” sub-discipline of their discipline to themselves. As the F1000Prime publication set is a collection of high-quality biomedical papers, it is expected that we find most readers in the disciplines of biology and medicine. We find strong connections between engineering, chemistry, physics, biology, and medicine as well as their rather high reader percentages. The connections of arts and literature to biology and medicine are much weaker. The disciplines arts and literature, engineering, materials sciences, and computer and information science have connections to many other disciplines, as the central location in the network indicates.

Data availability

Figshare: Mendeley reader counts for F1000Prime papers. Doi: 10.6084/m9.figshare.1301463 (Haunschild & Bornmann 2014).

Comments on this article Comments (0)

Version 2
VERSION 2 PUBLISHED 11 Feb 2015
Comment
Author details Author details
Competing interests
Grant information
Copyright
Download
 
Export To
metrics
Views Downloads
F1000Research - -
PubMed Central
Data from PMC are received and updated monthly.
- -
Citations
CITE
how to cite this article
Haunschild R and Bornmann L. F1000Prime: an analysis of discipline-specific reader data from Mendeley [version 1; peer review: 1 approved with reservations, 1 not approved]. F1000Research 2015, 4:41 (https://doi.org/10.12688/f1000research.6062.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?
Key to Reviewer Statuses VIEW
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Version 1
VERSION 1
PUBLISHED 11 Feb 2015
Views
83
Cite
Reviewer Report 31 Mar 2015
Stefanie Haustein, École de bibliothéconomie et des sciences de l’information (EBSI), Université de Montréal, Montréal, Canada 
Not Approved
VIEWS 83
The study combines data from F1000 and Mendeley to analyze the (self-reported) disciplines of Mendeley users saving articles recommended in F1000. Research questions lack focus and clarity, analysis and discussions are weak, conclusions are absent. This is mainly due to ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Haustein S. Reviewer Report For: F1000Prime: an analysis of discipline-specific reader data from Mendeley [version 1; peer review: 1 approved with reservations, 1 not approved]. F1000Research 2015, 4:41 (https://doi.org/10.5256/f1000research.6490.r7640)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 08 May 2015
    Robin Haunschild, Max Planck Institute for Solid State Research, Heisenbergstr. 1, 70569, Germany
    08 May 2015
    Author Response
    “The research questions lack clarity and the motivation of the study should not solely be based on the availability of datasets (Mendeley and F1000). It should be emphasized in how ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 08 May 2015
    Robin Haunschild, Max Planck Institute for Solid State Research, Heisenbergstr. 1, 70569, Germany
    08 May 2015
    Author Response
    “The research questions lack clarity and the motivation of the study should not solely be based on the availability of datasets (Mendeley and F1000). It should be emphasized in how ... Continue reading
Views
71
Cite
Reviewer Report 06 Mar 2015
Rodrigo Costas, Centre for Science and Technology Studies, Leiden University, Leiden, The Netherlands 
Approved with Reservations
VIEWS 71
This paper presents an analysis of F1000Prime recommended publications in combination with Mendeley users statistics. I think that in general terms the methodology is fine and the results are correct, having some descriptive interest. However, I have the following major ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Costas R. Reviewer Report For: F1000Prime: an analysis of discipline-specific reader data from Mendeley [version 1; peer review: 1 approved with reservations, 1 not approved]. F1000Research 2015, 4:41 (https://doi.org/10.5256/f1000research.6490.r7642)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 08 May 2015
    Robin Haunschild, Max Planck Institute for Solid State Research, Heisenbergstr. 1, 70569, Germany
    08 May 2015
    Author Response
    “The paper lacks in my view a solid justification of its research questions. The two research questions proposed in the paper (“are F1000Prime papers only read by people from biomedicine ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 08 May 2015
    Robin Haunschild, Max Planck Institute for Solid State Research, Heisenbergstr. 1, 70569, Germany
    08 May 2015
    Author Response
    “The paper lacks in my view a solid justification of its research questions. The two research questions proposed in the paper (“are F1000Prime papers only read by people from biomedicine ... Continue reading

Comments on this article Comments (0)

Version 2
VERSION 2 PUBLISHED 11 Feb 2015
Comment
Alongside their report, reviewers assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Sign In
If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password.

The email address should be the one you originally registered with F1000.

Email address not valid, please try again

You registered with F1000 via Google, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Google account password, please click here.

You registered with F1000 via Facebook, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Facebook account password, please click here.

Code not correct, please try again
Email us for further assistance.
Server error, please try again.