Keywords
F1000Prime, Altmetrics, Mendeley, paper, evaluation
This article is included in the Research on Research, Policy & Culture gateway.
F1000Prime, Altmetrics, Mendeley, paper, evaluation
Interest in the broad impact of research (Bornmann, 2012, 2013) has resulted in new forms of impact measurements. Traditional forms of impact measurements using bibliometrics only allow the measurement of impact on research itself. These new forms which have been named as altmetrics (abbreviation of alternative metrics) pretend to measure the impact of research on other areas of society (than research) by counting the mentions of papers in social media: “Alternative metrics, sometimes shortened to just altmetrics, is an umbrella term covering new ways of approaching, measuring and providing evidence for impact” (Adie, 2014, p. 349). As altmetrics, the number of readers (on Mendeley), mirco-bloggers (on Twitter), and other consumers of research using social media are counted. Although scientometrics research on altmetrics is still in a very early phase (comparable to research on bibliometrics in the 1970s), the use of these data in research evaluation is already an issue. For example, altmetrics is considered in the Snowball Metrics project (Colledge, 2014). This project compiled a set of clearly defined indicators which will be used by participating universities (mostly Anglo-American universities) for research evaluation purposes. It seems that altmetrics will be used in practice before scientometrics research has produced standards on their reliable, fair and valid application (Weller, 2015).
This study uses one of the most important sources for altmetrics data, namely Mendeley. Mendeley “claims 3.1 million members. It was originally launched as software for managing and storing documents, but it encourages private and public social networking” (Van Noorden, 2014, p. 126). Since data from Mendeley can be received by an Application Programming Interface (API) without any problems and the coverage of the scientific literature has been pointed out as high (Priem, 2014), Mendeley is a very attractive data source for the reception of research. “Mendeley records the number of users that have listed it [i.e. an article], describing them as readers, whether or not they actually read it. Presumably, listing an article in Mendeley tends to reflect that an article has been read or will be read in the future, although there is no evidence that this assumption is true” (Thelwall & Maflahi, in press).
In this study, we match Mendeley data with data from F1000Prime. F1000Prime is a database with biomedical papers and their reviews by peers. It is intended as a support tool for researchers to receive hints for the most important literature. Since it is not clear who actually reads the F1000Prime recommended papers, we investigated the disciplines of researchers (and other people) who have read these papers. We are mainly interested in two questions: are F1000Prime papers only read by people from biomedicine or are people from other disciplines also interested? Which disciplines read F1000Prime papers frequently or seldom together? The latter question will be answered by using social network techniques.
F1000Prime is a post-publication peer review system of papers from medical and biological journals. This service is part of the Science Navigation Group, which publishes and develops information services for the professional biomedical community and the consumer market. Papers for F1000Prime are selected by a peer-nominated global "Faculty" of leading scientists and clinicians. The Faculty members rate the papers and explain their importance. This means that only a selected set of papers from the biomedical area covered is reviewed, and most of the papers are actually not (Kreiman & Maunsell, 2011; Wouters & Costas, 2012).
The Faculty nowadays numbers more than 5,000 members worldwide, assisted by further associates, which are organised into more than 40 subjects. Members can choose and evaluate any paper of interest; however, "the great majority pick papers published within the past month, including advance online papers, meaning that users can be made aware of important papers rapidly" (Wets et al., 2003, p. 254). Although many papers published in popular and high-profile journals (e.g. Nature, New England Journal of Medicine, Science) are evaluated, 85% of the papers selected come from specialised or less well-known journals (Wouters & Costas, 2012). The F1000Prime database is regarded as a useful aid for researchers (and other people working research-oriented) to obtain indications of the most relevant papers in the biomedical area: "The aim of Faculty of 1000 is not to provide an evaluation for all papers, as this would simply exacerbate the ‘noise’, but to take advantage of electronic developments to create the optimal human filter for effectively reducing the noise" (Wets et al., 2003, p. 253).
Within the first half of 2014 the reference manager Mendeley provided a new version of its API. Some restrictions of the previous API were lifted. For example, the usage statistics were previously provided in relative terms and only for the top three entries (Haustein & Larivière, 2014). The new API provides results in absolute numbers and not only for the top three but for all entries. Mendeley provides access to the readership status (e. g. professor, postdoc, or student) and the distribution of the Mendeley readership across scientific disciplines as well as countries via the API. Those sets of data can be correlated with other information available about papers (e. g. citations or Twitter counts).
Before one can start to use the Mendeley API, one has to register as a Mendeley user. Afterwards, registration of the desired application is necessary (http://dev.mendeley.com). Authentication with the API is done via OAuth 2.0. The credentials are set during registration of the application.
We used R (http://www.r-project.org/) to interface to the Mendeley API. It seems to us that using other interfaces does not change the functionality or responsiveness, but we did not try to use other interfaces. Mendeley provides sample codes for Javascript, Python, R, and Ruby (http://dev.mendeley.com/code/sample_code.html), whereby all requests to the API use HTTP GET and POST requests. Therefore, we suppose that any other scripting or programming language may be used. The reply is sent in Javascript Object Notation (JSON).
We requested user statistics for the F1000Prime publication set (n = 114,582 papers) using the PubMedID and DOI between the 4th and 6th of December 2014. We observed seemingly random connection problems. Sometimes those problems occurred after a few hundred or a few thousand requests. The largest chunk of requests we were able to get through the API without connection problems consisted of 47,629 papers. This large number of records is contrasted with smaller chunks of requests (between 1,049 and 9,307 records).
Mendeley provides a breakdown of the user count into sub-disciplines. The possible values for disciplines and sub-disciplines can be obtained directly from the API via the GET /disciplines endpoint. Each discipline has a certain number of sub-disciplines. The sub-discipline “miscellaneous” occurs in every discipline. Each Mendeley user can select a discipline and a sub-discipline from a drop-down menu. This piece of information is not mandatory, like the user’s location.
Pajek is used to create the F1000Prime readership network (http://pajek.imfm.si/doku.php; de Nooy, Mrvar & Batagelj, 2011) applying the spring embedder of Kamada & Kawai (1989). For detecting communities in the common readership of F1000Prime recommended papers, we used the VOS Clustering algorithm (Waltman, van Eck & Noyons, 2010), which is available in Pajek. The aim of this algorithm is to provide further insights into the structure of the network (Milojević, 2014).
We found 6,263,913 Mendeley readers for the F1000Prime publication set. 99.9% (n=6,257,603) of them share their discipline and sub-discipline. This is a much higher percentage than those who share their geographical location (Haunschild, Stefaner & Bornmann, in preparation). For the F1000Prime publication set, the vast majority (74.94%) of Mendeley users is found in the “miscellaneous” sub-discipline of all disciplines. Therefore, we added up all the readers of all sub-disciplines for each discipline. The results of our study are presented in Table 1. Nine disciplines have at least 1% of the readers of the F1000Prime publication set. The remaining 16 disciplines have less than 1%. As expected, most readers (81.78%) of the F1000Prime literature assign themselves to the biomedical (sub-) disciplines. All other disciplines comprise the remaining 15.19% of the F1000Prime readership at Mendeley. The third largest readership is found in the discipline psychology which is related to medicine. After chemistry, which is also related to biology, five other disciplines show readership values above 1% within the F1000Prime literature. Those disciplines seem rather unrelated to the field of biomedical research, especially environmental sciences (according to Figure 1, see also the description below). 3.05% of the F1000Prime readers at Mendeley come from other disciplines (not shown in Table 1). The disciplines with most readers below the threshold of 1% are: social sciences (0.67%), mathematics (0.42%), electrical and electronic engineering (0.27%), education (0.22%), and materials sciences (0.21%).
Only disciplines are shown with more than 1% readers (sorted in decreasing order).
Network of F1000Prime recommended readers from arts and literature (AnL), astronomy and astrophysics (AsAs), biology (Bio), business administration (BuAd), chemistry (Chem), computer and information science (CIS), design (Des), earth sciences (ESci), economics (Eco), education (Edu), electrical and electronic engineering (EEE), engineering (Eng), environmental sciences (Env), humanities (Hum), law (Law), linguistics (Ling), management (Man), materials sciences (Mate), mathematics (Math), medicine (Med), philosophy (Phil), physics (Phys), psychology (Psy), social sciences (SoSc), sports and recreation (SpRe).
We also analyzed connections between the disciplines. These are shown in Figure 1. A paper which is read by Mendeley users of different disciplines (e.g. biology and physics) constitutes a connection between these disciplines. Therefore, a paper which is read by Mendeley users of the same discipline does not contribute to the network system, but a paper which is read by Mendeley users of many different disciplines contributes many connections to the network. The size of the vertices in Figure 1 reflects the numbers of readers for each discipline. The thicker and darker the edges between two disciplines, the more frequently they have read a F1000Prime paper jointly. The location of the discipline vertex also informs about the connectivity. The closer the vertex is located towards the center, the more connections to different disciplines are found. There are 25 disciplines and 300 links among those disciplines in the dataset. With a density of 1, the network is rather dense. The average node degree is 24. According to Figure 1, the strongest connection shows up between biology (Bio) and medicine (Med). The disciplines computers and information science (CIS), engineering (Eng), and chemistry (Chem) have rather strong connections to biology (Bio) and medicine (Med). The discipline arts and literature (AnL) shows a low amount of readers (0.15%, close to sports and recreation with 0.16%) as well as a good connection to other disciplines in the network.
The community detection algorithm detected two communities in the network with biology, medicine, engineering, chemistry, and physics as one community (yellow vertices) and all other disciplines as the other (green vertices).
The (sub-) discipline of Mendeley readers is self-assigned and not mandatory. Still, we found that a large share (99.9%) of F1000Prime paper readers at Mendeley share their (sub-) discipline. Most readers (74.94%) assign the “miscellaneous” sub-discipline of their discipline to themselves. As the F1000Prime publication set is a collection of high-quality biomedical papers, it is expected that we find most readers in the disciplines of biology and medicine. We find strong connections between engineering, chemistry, physics, biology, and medicine as well as their rather high reader percentages. The connections of arts and literature to biology and medicine are much weaker. The disciplines arts and literature, engineering, materials sciences, and computer and information science have connections to many other disciplines, as the central location in the network indicates.
Figshare: Mendeley reader counts for F1000Prime papers. Doi: 10.6084/m9.figshare.1301463 (Haunschild & Bornmann 2014).
Wrote manuscript: RH and LB
Data acquisition: RH and LB
Data processing: RH
Data analysis: RH and LB
Produced graphics: LB
Manuscript revision: RH and LB
Views | Downloads | |
---|---|---|
F1000Research | - | - |
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
Competing Interests: No competing interests were disclosed.
Competing Interests: No competing interests were disclosed.
Alongside their report, reviewers assign a status to the article:
Invited Reviewers | ||
---|---|---|
1 | 2 | |
Version 2 (revision) 08 May 15 |
read | read |
Version 1 11 Feb 15 |
read | read |
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Already registered? Sign in
The email address should be the one you originally registered with F1000.
You registered with F1000 via Google, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Google account password, please click here.
You registered with F1000 via Facebook, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Facebook account password, please click here.
If your email address is registered with us, we will email you instructions to reset your password.
If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance.
Comments on this article Comments (0)