The relationship between altmetric score with received citations

Today, in addition to citations and with the expansion of Background: social media, the use of altmetrics has gained attention as a tool necessary for evaluating the effects of scientific publications. The present study intended to monitor Iranian pediatrics articles, as one of the leading areas of scientific publications in Iran, between the years 2010-2016 using altmetrics and citation-metrics, and then evaluate the relationship between the altmetric score and number of received citations. This is a practical study of the analytical descriptive type and the Methods: research methodology is scientometrics. This research included 1332 research articles, review articles and conference articles in the field of pediatrics from Iran during 2010-2016, published in the Web of Science. Authors, year, journal and social media was determined in these articles. Data analysis was carried out using SPSS21 software and descriptive and inferential statistics (Kolmogorov-Smirnov test and Spearman correlation). A total of 1138 articles have citations and 256 articles had Results: altmetric activity. The results indicate a significant correlation among the articles’ altmetric scores and number of received citations. Among the data sources of altmetric score, mentions of articles in Mendeley, Twitter, and Facebook had the highest ranking. The number of times an article was read in Mendeley had a significant correlation with the number of citations. It seems that altmetrics better represent the impact of newer Conclusions: articles, while older articles had received more citations. In addition, a high number of reads in Mendeley correlates with received citations. However, Mendeley reads do not involve altmetric score calculation algorithms, and this should be implemented in the future.


Introduction
Evaluating the impact of scientific publications is carried out using various methods. "Scientific citations" are published for different sources, such as journals, books, articles, conference abstracts, and patents, and have been one of the most important evaluation tools for the impact of scientific publications. However, even though citations have various advantages, relying on the number of scientific citations can have disadvantages and there are various criticisms surrounding them. For example: • The long waiting time for review and publication results in the content becoming out of date, leading to a delay in scientific citations to those sources and hence analysis of the impact of the research; • Many of the evaluation methods based on scientific citations, such as impact factor are actually indices for journal evaluation and not the articles (Riyahi et al., 1995); • The type and field of citations is often not evaluated and it is not possible to become aware of the citation motives. For example, citations might be given to criticize or reject a certain theory; • The significance of the citing source in citation-metrics is not considered for most indices. For example, the value of citing a source in a letter to the editor, or a review article or research article is not the same; • While the review process, publication and citations of texts is very lengthy, many people prefer to use unofficial sources, such as online sources in order to publish the initial results of their research, which are not included in scientific citations (Wang et al., 2013); • Today, online communication and virtual scientific discussions have obtained a special place in scientific communities. Results of numerous research studies are used in these online sources through various means, while it is even possible that no citation is given to them by official sources (Erfanmanesh, 2015).
Consequently, there are weaknesses and existing deficiencies in methods based on citation only, and also the tendency towards using the online environment has become a means for the appearance of other criteria for scientific evaluation and analysis of the scientific process. Priem et al. states that today methods based on citations are not the only criteria for evaluation because citations only measures visible effects, while nowadays using Web 2 tools, circumstances have been provided for creating new measurement criteria, which make the inconspicuous results of research evident. Priem name it "altmetrics or alternative metrics" (Priem et al., 2012). Altmetric is a means for measuring the effect of scientific publications on the web, especially in social media, which has a more expansive audience. It is an easy access method with better capability for tracing the effects of publications in various forms of media (Piwowar, 2013).
Currently, Almetric.com is considered one of the most authentic databases for calculating the altmetric score of approximately five million articles. Briefly, in order to report the altmetric score, categorize and provide an up-to-date description of articles, this database considers the following sources: Evaluating the effects of scientific publications through official channels using scientific citations and on the Web using altmetrics each has their advantages and disadvantages. Until now, using methods based on citations and altmetrics in various scientific communities has been discussed to a great extent, and much research has been carried out for evaluating the relationship between these two methods. Since the use of social media has increased during the past few years, article coverage using altmetric scores has also increased during these years, and as a result it is better that scientific citations and altmetric scores complement each other and be evaluated more for recent articles (Costas et al., 2014) Evaluating the impact of scientific publications with citation-and alt-metric methods has been carried out thus far in various fields (Salajegheh, 2015), (Hassan & Gillani, 2016), (Costas et al., 2014), (Robinson-García et al., 2014). Pediatrics is among the topics of great regard in various countries, since this field, in addition to having an important role in promoting a country's ranking in the World Health Organization (WHO), assures children's wellbeing and is part of the country's development infrastructure. Iranian scientific publication in pediatrics has approximately one percent of the world's pediatrics publications, and also includes more than one percent of all scientific publications in Iran during the past five years in the Web of Science database. According to the WHO statistics, Iran's significant scientific activities and advances in the field of pediatrics has resulted in impressive development and improvement of health indices in this field (http://apps.who.int/gho/data/ node.country.country-IRN).
The present study is an attempt to monitor the impacts of Iranian pediatric articles as one of the leading fields of scientific publication in Iran with citation-metrics and altmetric methods, and hence obtain the relationship between altmetric score and number of received citations. More importantly, by accurately studying the number of times articles are mentioned in sources and media involved with the altmetric score, their relationship with the number of article received citations will be considered, and finally the correlation between the number of article mentioned in these sources with the altmetric score will be evaluated.
Objectives of the current study 1. Determine the amount of citations for Iranian pediatrics article; 2. Determine the altmetric score of Iranian pediatrics articles; 3. Determine the relationship between altmetric score and number of citations received for Iranian pediatrics articles; 4. Determine mentions of Iranian pediatrics articles in altmetric score data sources; 5. Determine the relationship between mentions in altmetric score data sources with number of citations received for Iranian pediatrics articles; 6. Determine the relationship between the number of mentions in altmetric score data sources with the altmetric score itself.

Methods
This is a practical study of an analytical descriptive type, and the research methodology is scientometric. The literature included was research articles, review articles, and conference abstracts (due to their importance in this field and scientometric studies) in the field of pediatrics published from Iran during March 2010 to September 2016, as found by the Web of Science database. This time period was used due to the prevalence of social media use during these years.
The sample included 1332 articles, which had been retrieved with the following procedure: 1. Pediatrics journals (120 titles; Supplementary File 1) were chosen using the Journal Citation Reports of 2015.
2. These journal titles were merged together in an advanced search of the Web of Science database using the OR operator and all of their articles were retrieved.
3. The retrieved articles were limited to the geographical region of Iran, 2010-2016, and type, including research article, review article, and conference abstract.
Altmetric data was found using the altmetric.com database, which requires a unique identifier for articles. The following steps were undertaken for finding the altmetric data of all articles: 1. All 1332 articles were initially added to Endnote and were evaluated using the DOI and PUBMED ID. Then, articles that had none of the aforementioned identifiers were searched through "find reference updates" button on Endnote to complete reference information and obtain their DOI or PMID.
2. In cases where identifiers were not found by this method, searches were done in Pubmed, Crossref, doi, and Scopus in an attempt to obtain their DOI or PM ID.
3. From 1332 articles, the aforementioned identifiers were obtained for 1138 articles. All 1138 articles having identifiers were searched using the Webometric Analysis software (v.2 beta student) and also altmetric.com API in order to find the altmetric score and details of the scores. A total of 256 articles had an altmetric score then the citations number of these 256 articles were outcomed using the HistCite software.
Data analysis was performed using SPSS v21 software. Descriptive and inferential statistics (Kolmogorov-Smirnov test and Spearman correlation) were calculated.

Overall findings
The findings indicate that of the 1332 articles evaluated, a total of 2879 citations were obtained, and the highest citation rate for an article was 37 citations. Among the 1332 articles, 609 articles obtained no citations. In total, 256 articles also had altmetric scores. Of these, the altmetric score ranged from 502 to 0.25. Articles obtained an overall score of 1023, and the altmetric score mean was about 4 points (standard deviation, 31). The reason for this high standard deviation was an article with an altmetric score of 502 among the articles. Evaluations indicate a positive correlation (P=0.018) among the altmetric score and number of citations obtained (Table 1).

Citations and altmetric relating to authors, journals and year
The findings suggest that among the articles, those by Bakhshayesh et al., Ajallouyean et al., and Rezai et al. had Table 4

Relationship between altmetrics and citations
By evaluating the detail pages of the articles' altmetric score, it was found that 256 articles were mentioned in Menedely, CiteU-Like, weblogs, mainstream media, Twitter, Reddit, Facebook, Pinterest, F1000, and Google Plus. The number of mentions to these articles in reference managers, Mendeley and CiteULike (which is regarded as mentions in social media and instead is called 'reads'), was overall 2606 times, and in social media, including weblogs, mainstream media, Twitter, Facebook, Reddit, Pinterest, F1000, and Google Plus, they were mentioned overall 918 times.
Our findings suggest that among the 1332 articles evaluated, 234 articles were read 2595 times in Mendeley and the maximum number of readings for an article was 124 times. There was a significant relationship between the number of reads in Mendeley with the citations received (R=0.3; P=0.00).
In CiteULike, eight articles were read 11 times, and the highest reading rate for one article was four times. There is was no significant relationship between the number of reads in CiteULike and the number of citations received (R=-0.09; P=0.14).
In scientific weblogs, 13 articles were mentioned 18 times, and the maximum number of mentions for one article in weblogs was twice. There was no significant relationship between articles mentioned in weblogs and the number of citations received (R=0.19; P=0.52).
In the mainstream media, eight articles were mentioned 48 times, for which the maximum mentions for one article was 41 times. There was no significant relationship between article mentions in the mainstream media and the number of citations received (R=0.08; P=0.84).
On Twitter, 222 articles were mentioned 705 times, for which the maximum mentions for one article was 242 times. There is no significant relationship between articles mentioned on Twitter and the number of citations received (R= 0.1; P=0.7).
On Facebook, 67 articles were mentioned 136 times, for which the maximum number of mentions for one article was 11 times. There is no significant relationship between articles mentioned on Facebook and the number of citations received (R=0.06; P=0.76).
Overall, the number of articles mentioned was very low in F1000 (four articles), Pinterest (two articles), Reddit (one article), and Google Plus (four articles). The number of mentions for all articles was only once. Thus, measuring their correlation with the number of received citations was disregarded. Table 5 shows the details of articles mentioned in each of the altmetric data sources.
Relationship between altmetric data sources and altmetric score In order to evaluate the effectiveness and relationship of each of the data sources in the altmetric score, the correlation between number of mentions and readers in altmetric data sources with the altmetric score was also evaluated. However, as it was previously mentioned, social media, including Google Plus, Pinterest, Reddit, and F1000, were eliminated from this correlation test due to the very low number of mentions. Findings indicate that the number of mentions in Mendeley, weblogs, Twitter, and Facebook have a positive relationship with altmetric score (R>0; P<0.05) ( Table 6).

Discussion
Articles published from Iran was about 1% of the published articles in the field of pediatrics in the world, and regarding the population of Iran (80 million; about 1% of the world population) it seems to be an adequate publication quantity. Regarding the initial research question, regarding received citations for Iranian pediatrics articles, our findings indicate that 1332 articles received 2879 citations (on average two citations), while over 40% of the articles received no citations, which suggests the unbalanced quality of articles. The findings indicate that among 1332 articles, only 256 had altmetric scores. The low number of articles with altmetric scores may be due to lack of coverage by the Altmetric.com database for many of the articles, such as those that did not have unique identifiers, coverage of only preferred and not all social media, and also difficultly in tracing and monitoring the impacts of the articles, due to the high amount of information found on the Internet.
The present findings indicate that there is a direct correlation between the altmetric score and number of received citations for articles. Therefore, it seems that altmetrics can also be considered as a tool alongside scientific citations for the evaluation of scientific publication impacts. The correlation between citations and altmetric scores has are been previously confirmed in various research, for example by Costas et al. The three top articles with the most number of citations were all written with the cooperation of more than five authors; two articles had international cooperation. Article that obtained half of the total altmetric score (502 from 1024 total score) was written with the cooperation of authors from around the world. Therefore, it seems that cooperative writing, especially international cooperation, can have a significant effect on the impact of scientific publications in the field of pediatrics in Iran. In a study by Erfanmanesh (2015), approximately half of the articles with the highest altmetric score were articles with international cooperation. Among the top ten articles with regard to the number of received citations and the obtained altmetric score, only one article was a duplicate publication. Otorhinolaryngology were the top best journals in evaluating the number of citations and altmetric score, could be a confirmation of the relationship between citations and altmetric score. On the other hand, with regard to the second and third ranking of journals, none of these were among the five journals receiving the highest number of citations and altmetric score, and it seems that the Iranian authors need to put greater effort in choosing appropriate journals to publish their works in the field of pediatrics.
Calculating the mean number of citations to article in each year shows that the years of 2010, 2011 and 2012 had the highest mean number of citations. Therefore, the highest altmetric scores obtained is approximately related to recent years, including 2016, 2013, and 2015. With regard to the fact that in the evaluated years, earlier years had higher citations and more recent years had higher altmetric scores, it could be stated that contrary to receiving citations, which are a long and time-consuming process, new publications have obtained higher altmetric scores. This issue can be due to the greater use of social media during recent years. On the other hand, it could be stated that new articles are discussed and cited in social media at a faster pace after publication and up-to-date topics have more advocates in social media. This issue was also discussed in the study by Erfanmanesh (2015). Therefore, regardless of citations, which are a better indication of the effectiveness of older articles, evaluation of new scientific publications may be better specified by altmetric score. Regarding the relationship between altmetric score and citations, which has been shown in this and previous studies, obtaining higher altmetrics scores by new publications could be a new and better outlook on the potential for these articles to obtain citations in the future. This would however require further investigation.
Findings indicate that pediatrics articles in Iran have been mentioned in 10 media platforms, including Mendeley, CiteULike, weblogs, mainstream media, Twitter, Reddit, Facebook, Pinterest, F1000 and Google Plus. Among these, the reads of articles in reference manager resources (Mendeley and CiteULike) is over two and half times that of articles in social media. The pediatrics articles from Iran were mentioned 918 times in social media, including Twitter, Facebook, weblogs, mainstream media, Pinterest, Reddit, Google Plus, and F1000. Over all social media platforms, Twitter alone had 76% of the mentions. In previous studies, the number of reads in Mendeley and mentions in Twitter were higher than the other sources (Araújo et al., 2015;Erfanmanesh, 2015;Hassan & Gillani, 2016;Robinson-García et al., 2014) and(Zahedi et al., 2014). The number of mentions to articles in Pinterest, Reddit, Google Plus and F1000 were very low, whereas the evaluation of their relationship with scientific citations was disregarded. In a study by Thelwall et al. (2013), the same instance occurred for Google Plus and Reddit.
Regarding the relationship between data sources used in altmetric score with received citations in Iranian pediatrics articles, findings show that the number of readers in Mendeley and the number of scientific citations obtained had a direct and significant relationship. In studies by Asadi et al. (2014) (2014), the relationship between the number of Mendeley reads and citations was confirmed. However, in the present study there was no significant relationship observed between the readers of CiteULike and scientific citations. In studies by Asadi et al. (2014); Bar-Ilan et al. (2012), andLi et al. (2012), the relationship between readers of CiteULike and number of citations was significant.
The altmetric.com database does not involve the number of Mendeley reads in the altmetric score calculation algorithm because this database cannot access user profiles in Mendeley, and user tendency towards specific author, journal, publisher or organization in Mendeley is not clear for this database (https://www.altmetric. com/about-our-data/the-donut-and-score/). In addition, it seems that with regard to the direct correlation between the number of readers of articles in Mendeley and received citations, which was also confirmed in previous studies (Asadi et al., 2014;Bar-Ilan et al., 2012;Erfanmanesh, 2015;Haustein et al., 2014;Li et al. 2012), and(Zahedi, 2014), in order to show the potential of Mendeley reads to represent the article's impact, it is better to involve an altmetric score calculation algorithm. It should be considered that with a high number of users and article reads in Mendeley, an individual effect is quite difficult to measure the general trend of an article. A significant relationship was not observed between the number of mentions in evaluated social media and the number of citations. Finally, for certainty, further investigations show that there is also no meaningful relationship between total mentions of articles in all social media and the number of citations. In a study by Xia et al. (2016), a significant relationship was found between the number of citations and mentions on Twitter. In addition, in a study by Thelwall et al. (2013), there was a significant relationship between citations and articles on Facebook, weblogs, and mainstream media.
In order to further evaluate the correlation, the number of mentions in social media with altmetric scores was also evaluated and it was specified that the number of articles mentioned on Twitter, Facebook, weblogs, mainstream media and Mendeley with altmetric score had a direct and significant relationship. This may confirm the fact that the general algorithm created for the altmetric score in altmetric.com database is accurate. However, this issue definitely requires further investigations at different points in time, due to the extensive area of social media, its increasing growth, and changes in society's approval of them.
In order to compare the ranking of Iranian pediatrics articles to other articles published by the same journals, the ranking of 0.43 for Iranian articles is indicative that the status of articles published from Iran are in the upper half of articles in pediatrics journals.

Conclusion
Today, with the increasing orientation of the public and researchers to social media, due to easy access, a wide-ranging audience, expansive relationship with other specialists, sharing of scientific publications and related discussions, disregarding the effects of the web domain in the evaluation of scientific publications does not seem to be correct. Based on the current research findings related to the direct relationship between altmetric score and scientific citations, as also seen in previous research, it seems that alongside scientific citations, altmetrics can also assist in better evaluation of the impact of scientific publications, especially for newer publications. Altmetric covers various indices and a wide range of social media and is updated at a higher speed. However, scientific citations are time-consuming and they are affected by the publishing source, thus it seems that evaluation of a new document and also articles that are published in unfamiliar journals, such as journals not indexed in authentic databases, is better performed with altmetrics. On the other hand, the expansive realm of social media makes it much more difficult to manipulate the altmetric score rather than manipulating the number of citations.
In this research and previous research, among readable media, Mendeley and among social media, Twitter and Facebook, have the greatest share of article mentions. According to this research and previous studies, the number of readers, especially in Mendeley, has a direct relationship with the number of citations, and it is better to be involved in the altmetric score calculation algorithm. The relationship between mentions in social media involved in altmetric score and the number of citations also requires more extensive evaluation.
Finally, by considering the extensive realm of Web2, its massive amount of data, the quick growth of social media, appearance of new facilities and new social media, and changes in people's orientation, presenting an appropriate strategy for calculating the altmetric score requires extensive research and should be kept up-to-date.

Zohreh Zahedi
Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, The Netherlands This paper has analyzed the correlation between citations received by WoS articles, reviews and conference papers in the field of Pediatrics and their altmetrics scores obtained from altmetric.com. It is a very descriptive study. It make sense to do such an analysis for the field of Pediatrics as one of major fields of scientific publication in Iran, according to the authors. However, there are lots of altmetrics studies that have used correlation analysis between altmetrics and bibliometrics indicators as this paper and hence this is not a new topic, doesn't contribute to the altmetrics literature and hence has no added value. It is not clear the importance of this study, lacking enough relation, in other words, what gaps is going to be filled and for what reasons? Also, to my idea, the paper needs major revisions in terms of English language (grammar, use of identifier such as the, etc. and not well written) and lacks sufficient justification of the study and interpretation of the results and other issues that are explained below. Hence, I don't suggest the paper for indexing due to these and the following reasons: The introduction needs stronger justification in terms of the aim of this study since correlation between altmetrics and citations has been done before in many studies and is not a new thing; The part including objective and aim of the study overlaps and repeated several times, the repeated sentences should be merged and summarized in one or two sentences.
The review of the literature on the correlation between altmetrics is not comprehensive and it misses some major studies which were done based on WoS, Scopus and PubMed databases across different fields. Additionally, the study by Thelwall et al. (2013) or the one by Haustein, et. al (2013) , or other relevant studies are missing here in which they have done the same correlation with all the indicators from altmetric.com across medical fields. It would be nice to compare the results of the above papers in terms of coverage, proportion, and correlation analysis, etc. with the result of the current study.
Regarding the following sentence the same holds true for altmetrics as well since motivation for 1 2 4. 5.
Regarding the following sentence the same holds true for altmetrics as well since motivation for tweeting scholarly papers is not known as well : "it is not possible to become aware of the citation motives. For example, citations might be given to criticize or reject a certain theory" I don't agree with the following statement, the sentence should be rephrased: "Since the use of social media has increased during the past few years, article coverage using altmetric scores has also increased during these years, and as a result it is better that scientific citations and altmetric scores complement each other and be evaluated more for recent articles " This sentence should be removed, it seems that it is a translation from Persian language and is not common to use it in the paper: "This is a practical study of an analytical descriptive type, and the research methodology is scientometric".
Regarding the methodology, the authors have used different identifiers. I wonder how the authors handle duplicates in terms of papers with more than one identifier? Please clarify it in more details in the paper.
Also, I would suggest to provide the % and details of data collection in a table. A descriptive table of the data set with citations, altmetrics, coverage, and percentage, etc. is lacking.
In the result section, I would suggest to add the proportion next to the raw values. For example, page 5: "it was found that 256 articles were mentioned in Mendeley…", better to add the % next to 256 articles both in the text and in a descriptive table that explained above.
"Outcomed" should be replaced with 'collected' in the following sentence: "From 1332 articles, the aforementioned identifiers were obtained for 1138 articles. All 1138 articles having identifiers were searched using the Webometric Analysis software (v.2 beta student) and also altmetric.com API in order to find the altmetric score and details of the scores. A total of 256 articles had an altmetric score then the citations number of these 256 articles were outcomed using the HistCite software".
please replace "evaluated" with "searched" using DOI: "All 1332 articles were initially added to Endnote and were evaluated using the DOI and PUBMED ID. Then, articles that had none of the aforementioned identifiers were searched through "find reference updates" button on EndNote to complete reference information and obtain their DOI or PMID".
Also regarding comparing citations with altmetrics score, comparing citations with individual altmetrics (tweet, Mendeley, etc.) is fine but with the aggregated altmetrics score is not recommended since it is a combined score of different altmetrics and doesn't include all metrics in the score, the weight is also different for different source.
To my idea it is better to use "received by" or other equivalents instead of "allotted" in the following sentence: "The highest number of citations was allotted to 2010, 2011 and 2013, and the highest altmetric scores were allotted to 2013, 2015 and 2016, respectively.
In the following section and some other parts, "evaluated" and "eliminated" should be replaced by "analysed or studied" and "excluded": "Relationship between altmetric data sources and altmetric score. In order to evaluate the effectiveness and relationship of each of the data sources in the altmetric score, the correlation between number of mentions and readers in altmetric data sources with the altmetric score was also evaluated. However, as it was previously mentioned, social 3 15. 16.

21.
with the altmetric score was also evaluated. However, as it was previously mentioned, social media, including Google Plus, Pinterest, Reddit, and F1000, were eliminated …".
"impacts" should be replaced by "impact".
The following statement is very strong and it is not true to mention that due to the existence of correlation, altmetrics can be used for evaluation disregarding challenges and quality issues for altmetrics data: "The present findings indicate that there is a direct correlation between the altmetric score and number of received citations for articles. Therefore, it seems that altmetrics can also be considered as a tool alongside scientific citations for the evaluation of scientific publication impacts".
The sentence should be rephrased: "Therefore, regardless of citations, which are a better indication of the effectiveness of older articles, evaluation of new scientific publications may be better specified by altmetric score".
Page 7: discussion, 'are' should be removed: "the correlation between altmetrics scores has are been ….).
I would suggest to distinguish between altmetrics score and individual altmetrics in the paper. Altmetrics score is the aggregated and combined score calculated by altmetric.com and which doesn't include all altmetrics in its calculation but individual altmetrics refers to reader counts, tweets, etc. Also, for Mendeley reader counts, it is better to use Mendeley API than altmetric.com for reader counts.
It is better to report the correlation coefficients like r=0.01 instead of 0/01. Also, to my knowledge, it is not common to report the p value for correlation in the table. Also, the authors need to specify the type of correlation analysis (Pearson or Spearman) they have used, due to the skewness of altmetrics and citations Spearman correlation analysis is suggested.

Are sufficient details of methods and analysis provided to allow replication by others? Partly
If applicable, is the statistical analysis and its interpretation appropriate? Partly Are all the source data underlying the results available to ensure full reproducibility?
No source data required Are the conclusions drawn adequately supported by the results? Partly No competing interests were disclosed.

Competing Interests:
I have read this submission. I believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com