Sentiment analysis of Indonesian tweets on COVID-19 and COVID-19 vaccinations

Introduction

Since first named as a global pandemic by the World Health Organization (WHO) in March 2020, COVID-19 has been the utmost issue challenging all aspects of human life worldwide.^1,2 Whilst, at the beginning of the crisis, little had been known about the pathogen, its detection, and its management; the disease began to increase the morbidity and mortality rate steeply and thus overwhelming the health care system in many countries, especially during the pre-vaccination period.³

One of the keys to controlling the morbidity and mortality rate increase is vaccination allied with the implementation of WHO-recommended efforts for reducing transmission, together with strong epidemiology surveillance.⁴ These are called for public literacy and engagement on COVID-19 and COVID-19 vaccination, which sentiments represent public opinion and beliefs toward the issues could be read on various platforms of social media including Twitter and might affect public acceptance toward the vaccine and vaccination program.^3,4 Although robust evidence has supported the efficacy and safety of various types of COVID-19 vaccines, public skepticism concerning vaccine effectiveness and side-effects has become a significant shortcoming in achieving wide-vaccination coverage.⁵

Analysis of social media content is a reliable way of mining for peoples opinions and beliefs, including toward COVID-19, COVID-19 vaccines, and vaccination; thus might help decision-makers to develop policies related to COVID-19 and COVID-19 vaccination.⁶ The use of social media has been rising drastically and the analysis of public opinion uploaded on social media can be an effective means to capture real-time public sentiment.⁷ Previous studies have been conducted on public opinion regarding COVID-19 vaccines and vaccinations in various countries. Sentiment analysis on tweet posts successfully portrayed public’s communication and perception on COVID-19 vaccines across India in the beginning of vaccine rollout, showing 44.65% positive sentiment of the Indian people towards ‘COVID-19 vaccines’.⁸ Analysis of global tweets also captured the greater impact of tweets expressing positive sentiment compared to those expressing neutral or negative sentiments.⁹

In Indonesia, hesitancy towards the vaccine and low literacy levels affect the acceptance of COVID-19 vaccination or certain vaccine brands.¹⁰ An early survey on vaccine acceptance from 1359 Indonesian respondents conducted from the end of March to April 2020, showed the acceptance rate as high as 95% for the free vaccine with reported efficacy of approximately 95%. However, the acceptance level dropped to 67% if the reported efficacy of the COVID-19 vaccine was only 50%.¹¹ The Ministry of Health of the Republic of Indonesia stated that a survey on vaccination by the COVID-19 Symptom Survey conducted by the University of Maryland Joint Survey Methodology Program in partnership with Facebook showed several factors cause doubts about vaccination acceptance amongst Indonesian people e.g. concern about the side effects, and the comorbidity that may affect the post-vaccination health state.¹² Although Indonesia has the fourth-highest number of social network users worldwide,¹³ studies on social media data to identify public sentiments and engagement towards COVID-19 and COVID-19 vaccination have still been limited and thus is the aim of the current study.

Method

Ethics

This study received approval from the Health Research Ethics Committee (KEPK), Faculty of Medicine, Universitas Airlangga, Indonesia (approval no. 145/EC/KEPK/FKUA/2021).

Several stages were carried out in this study to identify and analyze public opinion (Figure 1). The preprocessing started with data preparation which included determining searching terms for COVID-19 and COVID-19 vaccinations, then crawling the Indonesian-language tweets data based on the meta-tagging language stored on Twitter (https://twitter.com/?lang=en-id) during the period of 1^st January 2020 to 31^st August 2021.

Figure 1. The process stages of the dataset on Indonesia Tweets during 1st of January, 2020–31st of August, 2021.

The collected data were clustered into categories according to the search terms listed in Table 1. Subsequently, the data was cleaned from several elements such as emoticons, hashtags, non-alphanumeric characters, and URLs. The lower-text was used to convert all Twitter text to the lowercases. We then removed the punctuation in the Tweet text. The samples of dataset were taken from the cleaned data. The sample dataset was subjected to manual labeling within 3 categories, namely positive, neutral, and negative. We developed the sentiment model by using the manually labeled data and IndoBERT as tokenizer. The sentiment model was applied to run the labeling of all remaining data (Figure 1).

Table 1. Searching terms COVID-19, COVID-19 vaccine and vaccination.

Category	Searching Terms
	COVID-19
	COVID-19, corona, corona virus, corona virus, SARS COV-2, COVID, masks, keep your distance, physical distancing, social distancing, washing hands, PPKM, PSBB, lock down, WFH, LFH, online learning, self-isolation, swab, PCR, 3M, 5M, 6M, tracing, comorbid COVID-19.
	COVID-19 Vaccine and Vaccination
Vaccine Type	COVID-19 vaccine, Sinovac, Chinese vaccine, Nusantara vaccine, red and white vaccine, biopharmaceutical vaccine, inactivated vaccine, mRNA vaccine, Zeneca Astra vaccine, Pfizer vaccine, Moderna vaccine.
Vaccine Effectiveness	Immunity, prevent COVID transmission, COVID positivity rate, positive COVID, herd immunity, death rate; delta variant, delta strain, variant of concern, prevent severe COVID, prevent ICU needs, prevent MRS needs.
Vaccine Side Effect	KIPI, post-immunization co-occurrence, paralysis, blood clots, blood clots, vaccine death, death after the vaccine, allergy, positive for COVID-19, drowsiness, hunger, sexual disorders, vaccine side effects, vaccine hazard, stroke, Guillain-Barre syndrome, pain, swelling, dizziness, headache, fever, muscle aches, vaccine chip.
Vaccination	National COVID vaccination, national COVID immunization, vaccination acceleration, mass vaccination, health worker vaccination, elderly vaccination, child and adolescent vaccination, third dose vaccination, booster, vaccination stage 1, vaccination stage 2, vaccination stage 3 vaccination, vaccination of BUMN, vaccination certificate, vaccination fake, self-vaccination, vaccination age 12-17, vaccination age 18-59, cooperation vaccination, paid vaccination, vaccination comorbid.

The data was subjected to exploratory data analysis (EDA), in order to to find out the insight about the data, to discover patterns, to spot anomalies, and to check assumptions with the help of statistics and graphical presentations.¹⁴ The yield data was presented as the distribution of sentiment. The metric of tweet exposure was presented by word cloud and by the graph that depicted the dynamic changes by month within the captured period. The metric of tweet engagement involved the measurement of the metadata accompanying posts, including the number of retweets, likes, and replies. We also identified the circulating facts and the fake news on Twitter (Figure 1). Python visualization was used for data presentation.¹⁵

Results and discussion

A total of 3,489,367 Indonesian tweets data were collected from 1^st January 2020 to 31^st August 2021. The total dataset obtained was 324,358 data; consisting of 24,579 data related with COVID-19 and the rests were related with COVID-19 vaccine and vaccination.

Figure 2 showed the peak counts of tweets were seen in three parts during this period of time. In June 2020; in January 2021 and again in July 2021. The word-cloud of these tweets were seen in this figure, e.g. corona virus, health protocol, COVID-19 spread, preventing COVID-19, PPKM (abbreviation of Indonesian government program to control COVID-19), hand washing and physical distancing. Several keywords i.e. PCR (polymerase-chain reaction), antigen swab and transportation protocols dominated the tweets.

Figure 2. Tweets distribution on COVID-19 during 2020-2021. The peak counts were seen in June, 2020; in January, 2021; and in July, 2021 (graph). Word-cloud on COVID-19 were dominated by keywords of the virus, health protocols, detection methods and travelling protocols (words).

During 2020 when the vaccine and vaccination programs had yet known, the trending keywords were more into the COVID-19 virus and how to take a preventive step i.e. PPKM (pemberlakuan pembatasan kegiatan masyarakat)/restrictions towards community activity), and working from home/wfh (Figure 3). The word count graph shows many Twitter users discussed the COVID-19 virus along with tweets containing education to prevent exposure to the virus. During that period, the topic of discussion was still about education to suppress the spread of COVID-19. Preventive actions still being promoted, such as staying at home, avoiding direct contact with other people, avoiding non-essential travel, social distancing, frequent hand washing, and so on^20,21 remain hot topics among the public. Meanwhile, the word count shows a hot topic among the people after implementing PPKM. After the government established and implemented the PPKM policy, Twitter users discussed correlated issues such as working from home.^22–24 PPKM is intended as a form of response to the increase in COVID-19 cases, so the problem of the virus is still quite busy being discussed on Twitter. Implementing PPKM levels 1-4 by the government, which has brought pros and cons to the community, has also become a topic of discussion. Even so, this policy is considered effective in suppressing the surge in the increase in COVID-19 cases.²²

Figure 3. Trend keyword on the tweets showed that the public engagement and exposure during 2020 were predominantly on the virus, corona, COVID-19, PPKM and working from home/WFH, respectively.

In general, retweets dominate engagement, followed by replies (Figure 4). This supports evidence that the public is more focused on sharing information such as the rise or fall of cases as well as information on education and preventive measures implemented by the government to suppress the spread of COVID-19.²³

Figure 4. Engagement on COVID-19 were predominated by retweets during 2020-2021, except at the end of 2021 and in July, 2021 were the reply was higher than retweets and likes.

On the other hand, the tweets count on COVID-19 and COVID-19 vaccination was highest at July 2021 (Table 3), arguably due to the rise of SARS-CoV-2 delta variant that significantly increased the morbidity and mortality rate, thus called for more definitive prevention act.^23–25

The engagement on COVID-19 vaccine was topped by the likes during 2020-2021, followed by retweets and reply (Figure 5). In Indonesia, several brands that have gone through safe and halal tests that adapt to the conditions of Indonesia, being a country with one of the largest Muslim population in the world, are exciting topics to be discussed by the public.²⁴ For example, the most popular brand in Indonesia is Sinovac.²⁵ This is because Sinovac is the fastest vaccine brand to enter Indonesia.²⁶ It can be seen from the data that has been successfully presented that Sinovac continues to dominate, especially at the end of 2020 and early 2021, when the Sinovac vaccine has entered Indonesia. The Moderna vaccine also experienced an increase in the number of tweets because, during this period, many of these vaccines were distributed in Indonesia.²⁷ Moderna vaccine produces more side effects than other vaccines,²⁸ thus becoming increasingly discussed on Twitter. The AstraZeneca vaccine also carries more frequent side effects when compared to the Sinovac vaccine (Figure 5).²⁹

Figure 5. Engagement on the vaccine of COVID-19 during 2020-2021 was shown to be highest at July, 2021 by the likes, followed by retweets and reply, respectively (graph). The word-cloud of this topic was predominated by the kind of vaccine brands (words).

The side effect of COVID-19 vaccine was seen as the engagement topic topped by likes, retweets and reply which were higher in December, 2020; in March, 2021 and in July, 2021 compared to other months during this period. Whereas the type of vaccine engagement showed higher in February 2021; in May 2021 and in July 2021 compared to other months (Figure 6).

Figure 6. Engagement ratio on the side effect of COVID-19 vaccine was shown highest at June-July, 2021; followed by in March, 2021 and at the change of the year from 2020 to 2021 (top). The below chart showed the engagement level of vaccine type in this period, which topped by the likes and followed by retweets and reply, respectively.

The graphs and word cloud show that of the vaccine side effects subtopics recorded, the word most frequently discussed in tweets was COVID-19 vaccine and its side effect (Figure 7).

Figure 7. Word-cloud of vaccine side effects subtopics, predominated with the trend keywords e.g. Vaccine, COVID-19, positive COVID-19 and side effect.

On the other hand, the trending keywords on the vaccine effectiveness was topped by trending keywords of the disease transmission, management guidelines, virus variance e.g. delta variant also with the immune system, prevention and health system (Figure 8).

Figure 8. Word-cloud of the vaccine effectiveness subtopic based on the searching terms were dominated by the transmission of COVID-19, delta variant, strategy of prevention, management of the disease and preventive strategy.

From Figure 9, we can see that the paid vs. free vaccination programs became the trending keywords along with the vaccination program of the government and herd immunity. People were cautious on the paid vaccination, when in reality the vaccination program was held nationally since February 2021 by the government with primary target of healthy people aged 18-59 years old, and also prioritized for the health providers to instigate the herd immunity that was discussed as the ideal condition after vaccination coverage was achieved widely.^23–25

Figure 9. Word-cloud of the vaccination sub-topic, where paid vaccination, vaccination stage, government vaccination program and free vaccination became the trending keywords.

Based on the analysis using similarity value, 15 tweets were indicated as fake tweets with misleading content. Some of them include the worldwide approval of a dendritic cell-based vaccine candidate developed by a group of Indonesian researchers, which in fact had not even been authorized into phase 2 clinical trial by the Indonesian authoritative body. There was also content about the affiliation of several vaccine manufacturers to certain companies. Another topic was that up to 21% of vaccine trial participants experienced adverse effects after receiving the Moderna vaccine. The fake tweets of vaccine manufacturing companies presumed to have been developing COVID vaccines even before the COVID-19 pandemic started was also found to be misleading content. From the entire dataset, it can be concluded that there was extremely low tweet activity of spreading fake news with a negative context, such as finding supporters for fake news from tweets that are replied.

The sentiment referred to in the discussion includes users agreeing and understanding the conditions for COVID-19. The debate regarding the support for preventive actions by the government is also a topic that is still hotly discussed. In addition, Twitter users continue to carry out their activities as usual and provide education on how to prevent exposure to the virus, increasing positive sentiment about the issue. In the second period, the trend of positive sentiment led to discussions around the expressions of Twitter users to express their response to the increasing number of COVID-19 cases in Indonesia, accompanied by campaigns from all parties to carry out vaccinations aggressively. However, there was an increase in negative sentiment in the second period compared to the first period. This period was when the government begins to issue PPKM policies that reap the pros and cons of the community. PPKM, which has a level of 1-4, was arguably considered to harm the community’s economy because of the limited activities of the community at work. During PPKM 3-4 part of rules was closing the purchasing center at 20.00 GMT+7, and making the visitor capacity a maximum of 50%.³⁰ Thus, many parties have complained about the condition of the policy. The sentiment distribution every monthis presented as shown in Figure 10.

Figure 10. Public Sentiment about COVID-19 was predominated by the positive opinions, whilst the neutral and negative sentiments were followed, respectively by the approximately half of the prior sentiment (graph). The chart showed that the positive sentiment outweighed the negative sentiments and were peaked in June, 2020; in January, 2021 and in July, 2021.

Sentiment analysis was conducted to find out the positive and negative sentiments of the public towards the vaccination program in 2020, shows that positive tweet sentiment dominates all existing sentiments. Analysis of the dataset indicates that generally, tweets come from news accounts where tweets originating from these accounts are classified as neutral tweets. While the period of 2021, shows that although the number of tweets is more dominant than in the first period, it shows that public opinion has a positive sentiment tendency. This might be due to the public rise of awareness of the importance of vaccination to help reducing the spread of the COVID-19 virus. However, negative sentiment is still a significant problem because people expect only certain types of vaccines would work, and/or are still doubtful and do not believe in vaccination programs to tackle the pandemic (Figure 11).

Figure 11. (A) Vaccination sentiment analysis comparison between 2020 and 2021 (graph). The sentiment of COVID-19 vaccination program in 2020 (B) and 2021 (C) showed in the pie charts, respectively, where in both period the positive sentiment outweighed the negative sentiment by approximately 5 times.

Figure 12 shows that both in 2020 and 2021, positive sentiment tends to dominate above 50%. The proportion of positive sentiment decreases considerably while negative sentiment only slightly decreases in 2021. The percentage of neutral sentiment doubles in 2021 compared to 2020. Negative sentiment can happen because, in 2020, the existing vaccine research is still in the development stage by scientists, while during the first midterm of 2021, COVID-19 vaccines are still being rolled out to limited people in Indonesia.

Figure 12. The sentiment on the side effect of COVID-19 vaccine in 2020 and 2021 showed by the left and the right pie charts, respectively. The positive sentiment was higher than the negative or the neutral sentiments in both years.

In 2021, various vaccine products have gone through stages of trials and study results of their efficacies and side effects have been released to public by vaccine manufacturers. Due to the increasing clarity of reports regarding the effect of vaccine in stimulating the body’s immunity against virus, the public has more confidence in the role of vaccines in accelerating recovery from the pandemic. It is reflected by slightly increased positive sentiment of vaccination program in 2021 (Figure 11). However, fear toward vaccine’s side effect yet exists, possibly contribute to the decrease of positive sentiment in 2021 (Figure 12).

At Figure 13, the sentiment of the vaccination program in Indonesia was mostly positive (59.1%); whilst the analysis sentiment of various types of vaccines for COVID-19 that were available was predominated by positive opinions in general, during both years (approximately five times higher than the negative sentiments). We compared sentiment polarities toward vaccines in 2020 and 2021 and found increases in both positive and negative sentiments in 2021 after the vaccination program had started. Although positive sentiments showed a greater increase, the negative sentiments, which remained at 15.2%, deserve attention, as they may reflect a proportion of individuals with vaccine hesitancy and/or rejection.

Figure 13. The sentiment on COVID-19 various type of vaccines in 2020 (A) and 2021 (B) showed that the positive sentiments outweighed (C) the negative and neutral sentiments; the pie chart showed the public sentiment on the effectiveness of the vaccines in government vaccination program that was predominated by, again, positive sentiment followed by the neutral and negative sentiments.

This study shows tendency towards positive sentiments on COVID-19 topics and subtopics vaccine and vaccination throughout 2020 until mid-2021, somewhat contrasting with the results from other sentiment analysis on Indonesia tweets that showed predominantly neutral sentiment. These studies captured tweets within shorter period. Sumertajaya et al chose a time frame of January 15, 2021 to January 28, 2021, for the reason that this was the first period of the COVID-19 vaccination program being launched in Indonesia.³¹ While Agustiningsih et al. captured tweets within September 2021.³² Both used different method of learning framework. Sumertajaya et al implemented support vector machine (SVM) and random forest, while Agustiningsih et al. employed bidirectional Long Short-Term Memory (LSTM) combined with word embedding.^31,32

Social media studies have been used over the past decade to identify public opinion and sentiment toward particular health issues, for instance, the 2009 H1N1 outbreak.³³ The vaccination issue has been a matter of importance to be analyzed using this approach since the individual decision on vaccine uptake is modulated by opinions from social networks.³⁴ Disinformation narratives spread on social media that are sometimes hostile, causing anxiety, fear, and distrust toward vaccination could largely contribute to vaccine hesitancy and refusal.³⁵ The rapid and vast dissemination of disinformation should be addressed appropriately by effective strategies for vaccine promotion. The surveillance of real-time social media information flow could be a remarkable source of timely data updates for adjusting those strategies.³⁴ Interestingly, a strong correlation was shown between online-expressed sentiments and estimated vaccination rates.³⁶

Recent studies have confirmed evidence of vaccination impacts on several public health parameters, which give promise for its role in achieving herd immunity and further ending pandemic. Vaccination has remarkably reduced COVID-19 cases and hospitalizations.³⁷ COVID-19 related morbidity and mortality,³⁸ and incidence due to variants of concern.³⁹ Additionally, rapid vaccine roll-out is believed to boost economic recovery.⁴⁰ Therefore, accelerating the vaccination pace is imperative for countries worldwide.³⁷ Efforts to lower vaccine hesitancy should be prioritized as it is the greatest threat to achieving high vaccine coverage.⁴¹ Social media is perceived to be a major source of misinformation and is able to amplify and disseminate it without temporal and spatial limits, fostering vaccine hesitancy and lowering vaccine uptake.⁴² Our study identified a number of vaccine-related misinformation circulating on social media that were categorized as fake tweets. Our finding is in accordance with a study by Islam et al., that analyzed rumors and conspiracy theories related to COVID-19 vaccine on variable online platforms. They encountered that the majority of these contents were false and/or misleading. They also reported that Indonesia was among the countries with a high number of online rumors related to COVID-19 vaccine.⁴³ Our study found 15 vaccine-related fake tweets. Even though they were quite low in number considering the dataset collected was within 20 months, their impact on vaccine hesitancy shouldn’t be undermined.⁴⁴ We found fake tweets about the adverse effect of Moderna vaccine that was said to affect 21% of trial participants. This exaggerated misinformation certainly may raise public concern about vaccine safety, which may lead to vaccine hesitancy.⁴⁵ Another fake tweets issue was that vaccine manufacturers had been developing COVID-19 vaccines long before the pandemic emerged. This issue appeared to arise due to doubt about the fast pace of vaccine development. Some individuals may furtherly link it to the conspiracy theory that COVID-19 is a bioweapon designed by particular countries or parties. Meanwhile, belief in conspiracy theories is a driving factor for an individual to reject vaccination.^46,47 Actually, Twitter has set policies and conducted efforts to counteract any types of misinformation. It has launched detailed criteria and examples of false or misleading information about COVID-19 vaccine posted by users. Some actions taken to violations include content removal, tweet labeling, and adding corrective information. Twitter may also disable retweets, quoting, or any other ways of engagement to those false tweets, in case they pose potential harm to the public. The user’s account may be temporarily locked or permanently suspended.⁴⁸ It is very likely that Twitter has effectively cleansed any circulating misinformation. In addition, the ministry of health and the ministry of communication and informatics have provided platforms to continuously inform the public about the identified fake news or hoax and cooperate with social media to monitor misinformation related-contents and provide authoritative information as valuable trusted sources.⁴⁹ However, social media remains a battleground where the anti-vaccine movement is difficult to combat.⁵⁰

We deep-dived user-generated Twitter posts to elaborate on factors associated with vaccine hesitancy. Other than the spread of misinformation through social media, we discovered the public’s perceptions and concerns about vaccines effectiveness and side effects. The negative sentiment toward these topics may be associated with the public’s doubt regarding the vaccine as an effective and secure means to manage the pandemic. Our result was in line with a study on medical students reporting most of the participants had concerns regarding vaccines adverse effects and ineffectiveness.⁵¹ We identified a potential misconception regarding the side effects of the vaccine that it may indeed cause COVID-19 infection. This was actually one of the false myths identified and clarified by health authorities, yet still remains to be popular according to our findings. In this study, we found that people’s opinion showed on this particular social media is arguably correlated with their literacy on the issue.⁵² It might be affected by the ambience of social environment and information percepted at a certain period.

Despite the limitation that the collected opinion of Indonesian Twitter users might not fully reflect those of whole Indonesian population and that the used search terms might potentially cause overlap between the “COVID-19” and the “COVID-19 Vaccine and Vaccination” subsets, this study successfully confirmed the usefulness of social media studies to provide insights into the public’s attention, discussion, concerns, and sentiments about COVID-19 and COVID-19 vaccines and vaccination.⁵³ We demonstrated the dynamically changing public attention over time, where it peaked in July 2021, during the second surge of COVID-19 cases. It can be extrapolated that the public well responded to the government campaign of accelerating vaccine rollout as a means to curb the disease.⁵⁴ The top trend discussion topics may reflect a public concern, for instance, “paid vaccination. Even though we did not perform sentiment analysis on this specific topic, it can be proposed that the public highly disagreed with this policy. Since we did not perform time series analysis for public sentiment, our results only depict change of sentiment by year that hardly represent how sentiments have evolved over time month to month. Therefore, future studies should implement time series analysis to capture the dynamic nature of public opinion affected by the disease, circulating fake news or government actions. Beyond its potential usefulness, social media studies focusing on health issues, leave out several ethical concerns such as privacy, informed consent, and anonymity that remain the subjects of dispute.⁵⁵ Given the huge amount of dataset, informed consent was not possible to obtain in this study. However, anonimity is secured in this study, therefore the underlying data is not made published.

Conclusions

The public opinion and sentiment analysis on social media using an artificial intelligence of NLP may shortly provide timely data reflecting real-world public opinion. Thus, it should be part of the basis for developing strategies for public health response, particularly in a critical period of disease outbreak. Considering the high value of social media analysis, the more robust analytical methods should be used in the future studies, allowing for a clearer understanding of trends and patterns of public opinions of various health matters.