Influenza Altmetric Attention Score and its association with the influenza season in the USA

Background: Altmetrics measure the impact of journal articles by tracking social media, Wikipedia, public policy documents, blogs, and mainstream news activity, after which an overall Altmetric attention score (AAS) is calculated for every journal article. In this study, we aim to assess the AAS for influenza related articles and its relation to the influenza season in the USA. Methods: This study used the openly available Altmetric data from Altmetric.com. First, we retrieved all influenza-related articles using an advanced PubMed search query, then we inputted the resulted query into Altmetric explorer. We then calculated the average AAS for each month during the years 2012-2018. Results : A total of 24,964 PubMed documents were extracted, among them, 12,395 documents had at least one attention. We found a significant difference in mean AAS between February and each of January and March (p< 0.001, mean difference of 117.4 and 460.7, respectively). We found a significant difference between June and each of May and July (p< 0.001, mean difference of 1221.4 and 162.7, respectively). We also found a significant difference between October and each of September and November (p< 0.001, mean difference of 88.8 and 154.8, respectively). Conclusion: We observed a seasonal trend in the attention toward influenza-related research, with three annual peaks that correlated with the beginning, peak, and end of influenza seasons in the USA, according to Centers for Disease Control and Prevention (CDC) data.


Introduction
In the last few years, a new way to measure the attention brought by journal articles, termed altmetrics (a shortening of "alternative metrics" or "article-level metrics"), was adopted. It was also considered an "alternative" to the conventional citation-based measures. Altmetrics measure the impact and attention of an individual article 1 . Altmetrics are increasingly recognized tools with an aim to measure the real-time influence of an academic article 2 . Altmetrics measure the impact of journal articles by tracking social media, Wikipedia, public policy documents, blogs and mainstream news activity, after which an overall Altmetric attention score (AAS) is calculated for every journal article 3 . Altmetrics have been used to measure the impact of articles on a disease 4 , or even the impact of article on a whole field 5 . Altmetric.com is one of the providers of altmetrics and was found to have the best coverage of blog posts, news, and tweets. It pulls data from social media (e.g. Twitter and Facebook), traditional media (e.g. The Guardian and New York Times), blogs for individuals and organizations (e.g. Cancer Research UK), and online reference managers (e.g. Mendeley and CiteULike), policy documents published by official websites (e.g., .gov). The AAS is a quantitative measure of the quality and quantity of attention an output has received, it provides an indicator of the amount of attention a research has received. It weights the amount of attention received by each source based on an algorithm.
Each country has its own influenza detection center; the U.S has the Centers for Disease Control and Prevention (CDC), Europe has the European Influenza Surveillance Scheme (EISS), and Japan has the Infectious Disease Surveillance Center (IDSC) 6 . The problem of influenza detection and prediction can be tracked back to Serfling's work in 1963 in epidemiology, which tried to find a threshold for influenza breakout 7 . Since then, various approaches have been proposed for flu detection and prediction in multiple situations 7-9 . A previous project by Google in cooperation with the CDC was able to track in a population based on influenza-related web form queries on the Google search engine 10 . This approach has paved the way for many new approaches designed using the same concept of using search engines for flu detection in the USA 11 . In this study, we aim to assess the AAS for influenza related articles. Moreover, we will assess the top articles and journals publishing about influenza in terms of attention they brought.

Search strategy
This study used the openly available Altmetric data by Altmetric.com. Accordingly, this study was exempted from institutional board review IRB approval. We conducted the search on June, 5 th 2019. To retrieve all articles indexed in PubMed related to influenza, we used MeSH database to extract influenza-related terms, and the following were identified: • Grippe • Human Flu • Human Influenza • Influenza

• Influenza in Humans
We then searched PubMed database in the following steps: 1-All influenza entry terms mentioned above were used as "MeSH terms". It is important to note that the filter "Journal article" used in the search query only include original article, and exclude review articles, editorials …etc. We screened the searched results for articles discussing the use of databases to detect influenza in USA.

Altmetric data
We inputted the resulted search query into Altmetric Explorer, a web-based platform that enables users to browse and report all attention data for every piece of scholarly content. It provides the function of inputting search results already retrieved by the PubMed database 12 .
Data can be filtered and presented for countries and in specific time periods. We filtered influenza mentions for the USA as a country, to correlate with influenza frequency detected by the CDC, then we measured the AAS for each month in the period from 2012 to 2018, we then calculated the average AAS for each month.
We observed regular monthly mentions of the research output only after January 2012, thus we only included mentions from

Amendments from Version 2
In this revision, and as suggested by reviewers, we abandoned the use of "top 10" for tables in this article and reported number of instances according to a scientific explanation: For Table 2, we provided journals that published more than 100 research output related to influenza during 2019.
For Table 3, we provided research outputs with an Altmetric Attention Scores (AAS) of more than 1000 discussing influenza and their respective metrics.
Any further responses from the reviewers can be found at the end of the article REVISED January 2012 and on. We filtered the search for US mentions only. We collected US mentions of influenza related articles in each month in the years from 2012 to 2018, and we then calculated the average AAS score for each month.
We observe peak AAS scores, which defined as the highest score in a month compared to its previous and next months.

Statistical analysis
We used SPSS version 21.0 (Chicago, USA) in our analysis. We used mean (± standard deviation) to describe continuous variables (e.g. AAS). We used count (frequency) to describe other nominal variables (e.g. countries). We performed one-way ANOVA followed by Tukey's post-hoc test to analyze the difference in the mean AAS score between each month, we presented the results in mean difference with 95% confidence interval (CI). All underlying assumptions were met, unless otherwise indicated. We adopted a p-value of 0.05 as the significance threshold.

Results
A total of 24,964 PubMed documents were extracted. Among them, 12,395 documents had at least one Altmetric point. The total number of mentions for the included documents was 185,744, of which 152,899 were from social media, 20,499 were from news and blogs, 10,608 were from policy and patents, 1,309 were from other sources and 479 were from academic sources. The USA contributed to 28,001 (20.4%) of the total mentions, followed by UK 12,007 (8.8%), and Japan 8,684 (6.3%). The average US mentions for the influenza related articles each month from 2012-2018, and their total mentions are shown in Table 1.
On one-way ANOVA, we found a significant difference between the months (p< 0.001). Following post-hoc analysis, we found a significant difference in mean AAS between February and each of January (p< 0.001, mean difference of 117.4 with 95% CI: 89.7 to 145.2) and March (p< 0.001, mean difference of 460.7 with 95% CI: 430.2 to 491.1). We also found a significant difference between June and each of May (p< 0.001, mean difference of 1221.4 with 95% CI: 87.0 to 155.8) and July (p< 0.001, mean difference of 162.7 with 95% CI: 126.1 to 199.2). We also found a significant difference between October and each of September (p< 0.001, mean difference of 88.8 with 95% CI: 59.6 to 118.0) and November (p< 0.001, mean difference of 154.8 with 95% CI: 125.8 to 183.9). As shown in Figure 1, there are three peaks for the AAS; the highest is observed in February with a mean AAS of 1076.5 (±614.6), the second highest peak is in October with a mean AAS of 831.4 (±441.9), and the third peak is in June with a mean AAS of 586.2 (±271.1).
The journals publishing articles with highest AAS scores were PLOS ONE with a total AAS of 872 for 979 research outputs, followed by Vaccine with 842 for 1015 research outputs, and Influenza & Other Respiratory Viruses with 465 for 465 research outputs. Table 2 shows the journals that published more than 100 influenza-related research during 2019. The top research article in terms of AAS is entitled "Infectious virus in exhaled breath of symptomatic seasonal influenza cases from a college community" published in "Proceedings of the National Academy of Sciences of the United States of America" in January 2018, with an AAS of 2927. Table 3 shows research outputs discussing influenza by AAS with an AAS of more than 1000.
We found around 49 articles discussing the use of websites to detect influenza in USA ( Figure 2).

Discussion
The research on influenza attracted considerable attention, as measured by the AAS, with the USA the source of the greatest attention. For influenza research from the USA, we observed three peaks for the AAS. The highest peak is observed in February, with a mean AAS of 1076.5 (±614.6), which corresponds to the peak of influenza season as reported by CDC; the second peak is in October with a mean AAS of 831.4 (±441.9), which corresponds to the beginning of the influenza vaccination season; and the third is in June with a mean AAS of 586.2 (±271.1), which corresponds to the end of the influenza season. Almost 10,608 were from policy and patents, representing 5.7% of total attention score, which reflect influenza mentions in official and policy websites (e.g., websites ending with .gov).
Previous studies have used several analytic methods to correlate with influenza season. One of the first studies that brought significant public attention was the one that based its influenza surveillance on Google search engine query data 9 . in a study co-authored by Google Inc. and CDC researchers. The idea   In general, the interest in publishing about influenza has increased in the recent years 28 , with USA being the top country in terms of influenza research production 29,30 . From the overall influenza research output, influenza vaccine was one of the main topics researched and Journal of Virology and Vaccine journal published the highest number of research articles since 1900 29 . We also found that PLOS ONE was the top journal in terms of AAS followed by Vaccine.
With the recent emergence of the coronavirus (COVID-19), severe studies used the altmetric analysis to gain insight about its related publications and public's response to such new publications 31 . An altmetric analysis of COVID-19 articles found several factors affecting article's attention, including the title and how positive were the results 32 . Another article also showed a higher attention for articles published in high quality journals 33 . Such higher attention and impact of articles published in higher quality journals might be related to the dedicated social media centers in these journals to publicly promote published articles 34 .
Some limitations to the present study need to be taken into account. The search queries in these models are not exclusively submitted by users experiencing influenza-like symptoms, thus the correlations observed might be only meaningful across large populations. In addition, despite strong historical correlations, these systems remain susceptible to false alerts caused by a sudden increase in ILI-related queries. An unusual event, such as a drug recall for a popular cold or flu remedy, announcing a new flu strain, etc., could cause such a false alert 19 . Disease mentions sometimes depend on social events, which might not be related to disease spread, like holding a conference about flu pandemic. Another limitation to using web-based tools is coverage. Additionally, much of the world is currently excluded from the current systems, which can only process English-language tweets 20 . Future studies should further assess the validity of our descriptive results by performing sensitivity analysis using the number of articles published each year or proceeding years and correlate AAS score with weekly flu activity data. Moreover, other confirmatory studies may assess the validity of our results by assessing data for influenza in other countries such as Australia and New Zealand, where the influenza seasons are different in timing to those to the USA.
The use of social media interaction to describe epidemiological studies has been evolving. We observed a seasonal trend in the attention toward influenza-related research, with three annual peaks that correlated with the beginning, peak, and end of influenza seasons in USA, according to CDC data. We believe that analyzing the attention of influenza related research may aid in detecting influenza season's peaks, which may be a useful tool in areas with limited on-site detection centers. While this study is a descriptive data and its results provide preliminary data, its results should be cautiously interpreted due to the descriptive nature of the study.

Open Peer Review
in the tables or appendices? We know that there are errors like these in other publications, but we can not leave such errors repeating.
For example, Table 2 does not show the Journals stated in Table 3. The New England Medical Journal is stated three times among the ten mentioned in Table 3, but not in Table 2  Saif Aldeen AlRyalat, University of Jordan, Amman, Jordan Thank you for your productive comments throughout the review process. We agree with the reviewer that choosing "top 10" in general has no scientific basis, rather a haphazard number commonly used by research. Accordingly, we abandoned the "top 10" in this article and reported according to a scientific explanation. For table 2, we provided journals that published more than 100 research output related to influenza during 2019.
For table 3, we provided research outputs with an Altmetric Attention Scores (AAS) of more than 1000 discussing influenza and their respective metrics.

Dental Hypotheses, Isfahan, Iran
Keywords should be revised. General terms such as "detection" and "CDC" are not useful. 1.
Descriptions about Altmetric resources must be removed from the methods and could be presented in the introduction section.

2.
"It pulls data from: Social media (e.g. Twitter and Facebook). The main weakness of this manuscript is its statistical methods. The authors aimed to assess the correlation between two longitudinal data "We filtered influenza mentions for the USA as a country, to correlate with influenza frequency detected by the CDC". Yet do not perform any statistical analysis in this regard. I suggest the usage of https://play95.shinyapps.io/Repeated_Correlation/ 1. 3.

Figure 2.
This is a finding out of the aims and methods of this study. Please describe methods to reach this data in detail. Why is this important? Presentation of a new findings in the discussion section is not recommended.

4.
152,899 were from social media. Please describe which social media. Analyze active Twitter accounts in this regard.

5.
10,608 were from policy and patents. This finding is very important. Please describe policy resources.

6.
Please describe the limitations of the study and suggestions for future research in the discussion section.

7.
What is the clinical relevance of the findings of this research? 8.
Usage of new methods e.g. keyword co-occurrence network analysis or co-citation network analysis is popular among altmetric papers. The authors could consider these methods to increase the values of their research.

9.
Is the work clearly and accurately presented and does it cite the current literature? assess the correlation between two longitudinal data "We filtered influenza mentions for the USA as a country, to correlate with influenza frequency detected by the CDC". Yet do not perform any statistical analysis in this regard. I suggest the usage of https://play95.shinyapps.io/Repeated_Correlation/ Response: Thank you for the suggestion. Our study is mainly a descriptive analysis of the attention scores. We agree with the reviewer that such studies have a strong limitation in its design, which usually does not support inferential statistics. We edited the conclusion and further elaborated on the limitations so that reader keep this limitation in mind upon interpreting the results. Table 2. To my knowledge, Altmetric institution does not provide Altmetric Attention Score for journals. The only number of mentioned articles and total mentions calculate at the journal level.

1.
Response: The Altmetric database analyze the attention score at the article level as the reviewer mentioned, and they accordingly calculate the journal level AAS from their respective articles. to the USA." expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Department of Pediatrics, Pennsylvania State University College of Medicine, Hershey, PA, USA I commend the author for carrying out this important study and a very interesting study premise. However, there are a few points that need to be addressed.

Major comments:
Introduction: The entire second paragraph is irrelevant to the current study. Most of the literature cited in the second paragraph is about flu surveillance activity based on objective hospital data such as hospital deaths, pharmacy use, confirmatory laboratory test. Authors should look for literature about social media use or other similar findings to justify their study. Citation 10 and 11 are appropriate. This paragraph does not support the problem at hand nor justify the study premise.
Method: The author has written this aim in the introduction, "we aim to assess the AAS for influenza related articles and its relation to the influenza season in USA. Moreover, we will assess the top articles and journals publishing about influenza in terms of attention they brought.". Methods provided can justify the second objective but not the first one. Authors need to carry out sensitivity analysis to correlate AAS score and flu vaccine. For example, CDC releases weekly data about flu activity, author can show the correlation of that particular week with AAS score. In the current method, the author can state that this is a descriptive study of AAS during the flu season in the last couple of years. Author can not fulfill the first objective without sensitivity analysis and correlation with flu activity.
Results: This entire paragraph should be in the method section, "We observed regular monthly mentions of the research output only after January 2012, thus we only included mentions from January 2012 and on. We filtered the search for US mentions only. We collected US mentions of influenza related articles in each month in the years from 2012 to 2018, and we then calculated the average AAS score for each month. This is shown in Table 1." Basically, author is describing what they did for the study. I am not sure Table 1 is necessary. You can add standard deviation of table 1 in Figure 1. It is a duplication of information.
I have trouble understanding results, particularly this section. "As shown in Figure 1, there are three peaks for the AAS; the highest is observed in February with a mean AAS of 1076.5 (±614.6), the second peak is in October with a mean AAS of 831.4 (±441.9), and the third is in June with a mean AAS of 586.2 (±271.1)." What do authors mean by "peak"? As far as I can see from Figure 1, the mean AAS score was highest in February, January, October, September, December in that order. Why the author stated that the second peak was in October and then 3rd peak was in June. It seems figure 1 and the reported result are not correlating. What does "peak" mean? Please clarify. It seems authors trying to fit their data with CDC surveillance data.
Author has mentioned this in Method, " We filtered influenza mentions for the USA as a country, to correlate with influenza frequency detected by the CDC, then we measured the AAS for each month in the period from 2012 to 2018, we then calculated the average AAS for each month." The cited material here showed that flu activity peaked in February and there is minimal to no activity in April, May, and rest of the summer. If the author is stating that their findings help for flu activity surveillance then accordingly their AAS score should be minimal during those months but instead, the author reported "peak" in June.
It is not surprising to see higher AAS during flu season in US. The bottom line is "study findings are overstated." The objective and conclusion of the study are not supported by methods use and reported results, respectively. I would advise authors to turn it around and report this study as a descriptive study of AAS during flu seasons.
Based on the method and results, it is difficult to justify that author's findings will help CDC and/or other surveillance agency to monitor flu activity. For that, authors need 1) to do sensitivity analysis using the number of articles published each year or proceeding years and correlate AAS score (increasing or decreasing) with weekly flu activity data, 2) to choose denominator such as the number of articles published preceding years or something like that rather than just mean AAS score. They need to think out of the box for this, 3) finally, validation of the findings.
Discussion: Again, the author mention other surveillance studies but they have mostly reported pharmacy data, hospital, ED visits, and lab data. This should change according to method and results.
Authors need to make major revisions to the article before it can be accepted for publication.

If applicable, is the statistical analysis and its interpretation appropriate? Partly
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? No
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Trends in outcome data, Surfactant protein A I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 08 Jan 2022 Saif Aldeen AlRyalat, University of Jordan, Amman, Jordan I commend the author for carrying out this important study and a very interesting study premise. However, there are a few points that need to be addressed.

Major comments:
Introduction: The entire second paragraph is irrelevant to the current study. Most of the literature cited in the second paragraph is about flu surveillance activity based on objective hospital data such as hospital deaths, pharmacy use, confirmatory laboratory test. Authors should look for literature about social media use or other similar findings to justify their study. Citation 10 and 11 are appropriate. This paragraph does not support the problem at hand nor justify the study premise. Response: Thank you for the suggestion. We omitted the irrelevant part of the introduction, so that only relevant information on influenza detection kept to be more relevant to the current study.
Method: The author has written this aim in the introduction, "we aim to assess the AAS for influenza related articles and its relation to the influenza season in USA. Moreover, we will assess the top articles and journals publishing about influenza in terms of attention they brought.". Methods provided can justify the second objective but not the first one. Authors need to carry out sensitivity analysis to correlate AAS score and flu vaccine. For example, CDC releases weekly data about flu activity, author can show the correlation of that particular week with AAS score. In the current method, the author can state that this is a descriptive study of AAS during the flu season in the last couple of years. Author can not fulfill the first objective without sensitivity analysis and correlation with flu activity. Response: We agree with the author on the importance of sensitivity analysis. Due to the absence of such analysis, we modified the aim to be in line with the conducted analysis: "In this study, we aim to assess the AAS for influenza related articles. Moreover, we will assess the top articles and journals publishing about influenza in terms of attention they brought" Results: This entire paragraph should be in the method section, "We observed regular monthly mentions of the research output only after January 2012, thus we only included mentions from January 2012 and on. We filtered the search for US mentions only. We collected US mentions of influenza related articles in each month in the years from 2012 to 2018, and we then calculated the average AAS score for each month. This is shown in Table  1." Basically, author is describing what they did for the study. I am not sure Table 1 is necessary. You can add standard deviation of table 1 in Figure 1. It is a duplication of information. Response: We agree with the reviewer that this part of results is better fitted within the methods. We made the required changes. Table 1 also mentions the total mentions, which is used in the discussion of our results.
I have trouble understanding results, particularly this section. "As shown in Figure 1, there are three peaks for the AAS; the highest is observed in February with a mean AAS of 1076.5 (±614.6), the second peak is in October with a mean AAS of 831.4 (±441.9), and the third is in June with a mean AAS of 586.2 (±271.1)." What do authors mean by "peak"? As far as I can see from Figure 1, the mean AAS score was highest in February, January, October, September, December in that order. Why the author stated that the second peak was in October and then 3rd peak was in June. It seems figure 1 and the reported result are not correlating. What does "peak" mean? Please clarify. It seems authors trying to fit their data with CDC surveillance data. Response: We provided a definition to the "peak" we used in the results section, which is the relatively higher AAS compared to previous and next months. Accordingly, the highest is observed in February with a mean AAS of 1076.5 (±614.6), the second highest peak is in October with a mean AAS of 831.4 (±441.9), and the third peak is in June with a mean AAS of 586.2 (±271.1). The months mentioned by the reviewer has highest absolute values.
Author has mentioned this in Method, " We filtered influenza mentions for the USA as a country, to correlate with influenza frequency detected by the CDC, then we measured the AAS for each month in the period from 2012 to 2018, we then calculated the average AAS for each month." The cited material here showed that flu activity peaked in February and there is minimal to no activity in April, May, and rest of the summer. If the author is stating that their findings help for flu activity surveillance then accordingly their AAS score should be minimal during those months but instead, the author reported "peak" in June. Response: We agree that the AAS data is not completely explained by the CDC data related to infection cases. The small peak in June was hypothesized to be related to the end of influenza season. This is further stressed in the limitations.
It is not surprising to see higher AAS during flu season in US. The bottom line is "study findings are overstated." The objective and conclusion of the study are not supported by methods use and reported results, respectively. I would advise authors to turn it around and report this study as a descriptive study of AAS during flu seasons. Response: We agree with the reviewer that the AAS were not completely correlating with the CDC data, and the results should not over-emphasize the findings. We further elaborated on the study limitations and re-worded the conclusion to be in line with these limitations. Response: We further elaborated on the study limitations and conclusion to emphasize on the descriptive nature of the study and its findings.
Based on the method and results, it is difficult to justify that author's findings will help CDC and/or other surveillance agency to monitor flu activity. For that, authors need 1) to do sensitivity analysis using the number of articles published each year or proceeding years and correlate AAS score (increasing or decreasing) with weekly flu activity data, 2) to choose denominator such as the number of articles published preceding years or something like that rather than just mean AAS score. They need to think out of the box for this, 3) finally, validation of the findings. Response: Thank you for the great suggestions. We added the suggestions to as what future studies should do to confirm our descriptive results.
Discussion: Again, the author mention other surveillance studies but they have mostly