The impact of the open-access status on journal indices: a review of medical journals

Background: Over the past few decades, there has been an increase in the number of open access (OA) journals in almost all disciplines. This increase in OA journals was accompanied an increase in funding to support such movements. Medical fields are among the highest funded fields, which further promoted its journals to move toward OA publishing. Here, we aim to compare OA and non-OA journals in terms of citation metrics and other indices. Methods: We collected data on the included journals from Scopus Source List on 1 st November 2018. We filtered the list for medical journals only. For each journal, we extracted data regarding citation metrics, scholarly output, and wither the journal is OA or non-OA. Results: On the 2017 Scopus list of journals, there was 5835 medical journals. Upon analyzing the difference between medical OA and non-OA journals, we found that OA journals had a significantly higher CiteScore (p< 0.001), percent cited (p< 0.001), and source normalized impact per paper (SNIP) (p< 0.001), whereas non-OA journals had higher scholarly output (p< 0.001). Among the five largest journal publishers, Springer Nature published the highest frequency of OA articles (31.5%), while Wiley-Blackwell had the lowest frequency among its medical journals (4.4%). Conclusion: Among medical journals, although non-OA journals still have higher output in terms of articles per year, OA journals have higher citation metrics.


Introduction
Open access (OA) journals allow free (access to/availability of) academic articles, they enable any user to read, search, download, share, use them for indexing, print the full texts, or utilize them as data for software without being charged 1 . Over the past 20 years, there has been an increase in the number of OA medical journals. According to Web of Science, published OA articles as a proportion of total publications increased from 9.5% to 24% from 1998 to 2018. These OA journals provide an easily accessed source of information, a source that is accessible even for developing and low income countries 2 .
Bibliometric analysis are methods or applications used to measure the influence of authors or scientific papers, of which, citation analysis is the most commonly used methods 3 . Now several citation databases have become available, with the three largest being Web of Science, Scopus and PubMed. These databases record the number of times that a journal article has been cited by other papers 4 . The use of bibliometric analysis is becoming more popular to assess the performance of different aspects of the scholarly and scientific fields. Analysis can be at the level of the researchers themselves, journals, departments, universities, national organizations, and even entire nations 5-8 . There are several databases that can be used to perform the bibliometric analysis, with each database having its own characteristics; these include Google Scholar, Pubmed (Only biomedical citations), Scopus, and Web of Science 4 . According to the number, coverage, and quality of citations covered by the databases, Scopus has wide coverage of high quality journals, compared to high number of citations at the expense of quality for Google Scholar, and high quality at the expense of number of citations for Web of Science 9-11 . It is claimed that the emergence of OA journals has led to better dissemination of knowledge with the additional benefit of more citations for the authors, although this is still a matter of debate 12 . In this study, we aim to study the OA status of medical journals and the impact of the open-access status on journal indices using the Scopus database.

Data collection
We collected data on the included journals from Scopus Source List on 1 st November 2018 (see Underlying data 13 ). We filtered the list for medical journals (which include all specialties in medicine, as per Scopus categorization).

Variables
For each journal, we extracted the following citation metrics: Citation count, Percent Cited, CiteScore, CiteScore Percentile, SCImago Journal Rank, Source Normalized Impact per Paper (SNIP), and SCImago Quartiles. Details about these metrics and how they are calculated can be found on Scopus website.
Moreover, scholarly output is defined as sum of documents published in the serial title (e.g. 2017) in the 3 years prior to the year of the metric (e.g. 2014 -16). Open access Journals covered by Scopus are indicated as Open Access if the journal is listed in the Directory of Open Access Journals (DOAJ) and/or the Directory of Open Access Scholarly Resources (ROAD).

Statistical analysis
We used SPSS version 22.0 (Chicago, USA) in our analysis. We used means (± standard deviation) to describe continuous variables (i.e. journal indices). We used counts (frequency) to describe other nominal variables (i.e. publishers and OA journals). We performed Mann-Whitney tests to analyze the difference between measurements and OA status, and we presented data as medians (25% to 75% quartiles). To analyze open access journals between radiology and medicine, we used the weighting cases function in SPSS and a Chi-square test. All underlying assumptions were met, unless otherwise indicated. A p value of 0.05 was considered as significant.
Upon analyzing the difference between medical OA and non-OA journals, we found significant differences in the following indices: Upon comparing open access journals between the 5 most common publishers, we found a significant difference (p< 0.001).
Post-hoc analysis showed that Wiley-Blackwell has significantly lower number of open access journals 16 (4.4%) open access journals compared to others. Table 2 shows the open access status for the most common publishers.  We found that the number of OA journals varied among publishers, with Whiley-Blackwell having the least, with only 16 journals (4.4%), and the most with Springer Nature (206, 31.5%).

Discussion
In a previous study that analyzed OA articles published by different publishers, regardless of the discipline, they found that Elsevier had the highest number of OA articles, followed by Springer Nature and Whiley-Blackwell 22 . A longitudinal study comparing hybrid open access articles between publishers found great variation depending on the discipline 23 . For instance, medicine is the discipline which most frequently publishes in hybrid OA 23 .
Our study has potential limitations. In this study, we didn't account for the effect of publishing OA articles in non-OA journals (hybrid journals), as "Gold" OA publishing (i.e. fully OA journals) relates to publication of articles that are freely available to view and these may occur in OA or hybrid journals. Moreover, future studies should consider analyzing specialties within medicine (e.g. oncology), where we believe there will be variations in the effect of OA publishing within these specialties.

Grant information
The author(s) declared that no grants were involved in supporting this work. 1.

2.
3. General comment: This article describes difference of journal citation metrics between OA and non-OA journals in medical scope. The subjected journals are 5835 medical journals which are indexed in the Scopus and compared indices are metrics data supplied by Scopus and SCImago. The data showed significantly higher citation indices in OA journals than those in non-OA journals. The conclusion is that OA medical journals are cited more. The conclusion is known already in OA journals of other disciplines but it is valuable to publish.

Open Peer Review
The conclusion is sound and positive as expected, but I think we have to think about other issues on this topic. OA means no financial barrier for readers, which can facilitate citations. But most of researchers in established institutions have no barrier to non-OA journals because their institutions are able to subscribe to the non-OA journals. For those researchers, non-OA journals are free. I think that is the main reason of the small difference of those indices between the two groups although the difference is statistically significant. The statistics must be supported by large number of subject journals. This point must be discussed.

Specific comments
Definition of OA: journals included in DOAJ or ROAD. There are more Scopus journals outside of the 2 directories. PubMed Central may add more OA journals. The core results are difference of the indices that must be presented in a Table. Tables 1 and 2 are supportive but not cores. For reader friendly writing, one Table is required for results described in the text. Table 2 analyzed total numbers of OA journals by publishers. It may be interesting for readers to summarize whether there are index differences by publishers. Add discussion on the difference of indices: CiteScore 0.19 vs. 1.06; Scholarly output median 157 vs. 205; Percent cited 52% vs. 48%; SNIP median 0.706 vs. 0.617. All are highly significant by p<0.001. The gaps are significant but not so big because there are many other factors of citation. We should approach more details of citation analysis.

Is the study design appropriate and is the work technically sound? Yes
Are sufficient details of methods and analysis provided to allow replication by others? 1.

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? Yes The article is generally well done with respect to its method and purpose. My major concern is its purpose: assuming adequate methods, statistical analyses and result, what is the significance? If there are implications of the research or significance of the conclusions, they should be clearly stated. The article is dependent upon the accuracy and scope of the Scopus database. I wonder how that was chosen and whether Pub Med might have been consulted as well. The accuracy of the conclusions is no better than the comprehensiveness of Scorpus. Are articles that appear in print but are archived online included? Otherwise the study seems to have been conducted well (methodologically) and the results competently reported.

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? performance of parametric tests. I would like to see that they applied at least an ANOVA comparing three groups: OA, non-OA and OA articles in non-OA journals (hybrid journals). The hybrid group is fundamental, and they do not include it. The authors should mention to the reader the bias of this decision. Also in this point, it is essential the authors explain why, if previous studies have concluded that the Eigenfactorscore is the best predictor of citations , they did not include this metric in their analyses? Please also explain to the readers, why a linear-mixed-model design analysis was not performed. It is necessary to mention references of recent articles citing the existing correlations between the selected bibliometrics (CiteScore vs SNIP, Citescore vs IF, etc.) with at least two purposes: that the authors justified why they did not include a correlation analysis in their study, and that the readers be aware of the limitations in the correlation analysis, and also how the medical speciality may influence the results. It would be desirable to present a subgroup analysis of the medical specialities with the higher number of citations (for example oncology) as an example of the expected variability within subspecialties. If you report in the methods section that you used the SCImago Quartiles, why not control the effect of this variable using ANCOVA O MANCOVA? For example, if the authors are using the data from 5835 medical journals, this data allows a more robust analysis besides descriptive statistics and Mann-Whitney tests.

Are the conclusions drawn adequately supported by the results? Yes
No competing interests were disclosed.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com