Sentiment analysis of internet posts on vaccination using ChatGPT and comparison with actual vaccination rates in South Korea

Introduction

The exchange of opinions and information on social media platforms has become vital for social interaction and communication. This transformation has created an environment in which information and opinions about societal issues spread rapidly and are shared.¹ In recent years, the global landscape has been significantly affected by the COVID-19 pandemic, leading to an abundance of Internet posts discussing various aspects of this crisis. Among these critical topics, vaccination has emerged as an indispensable tool for overcoming the challenges posed by the pandemic.² Despite their importance, the distribution and vaccination rates vary significantly across nations, regions, and occupations,³ and these differences are closely related to individuals’ perceptions of and attitudes toward vaccines.^4,5

Gathering public opinion on vaccination through Internet posts is anticipated to contribute to providing accurate information and obtaining insights for policy formulation in infectious disease prevention.⁶ Sentiment analysis is commonly employed to analyze extensive and diverse online content. It is a form of text classification that focuses on subjective statements and is often called opinion mining. The goal was to analyze opinions to gain insights into public perceptions.⁷ However, the complexity and necessity of meticulous processes to enhance accuracy pose challenges for sentiment analysis.⁸ Moreover, the language used on social media platforms can vary in style among users and may require understanding within the context. In such cases, existing sentiment-analysis models may find it challenging to adapt to diversity and changes.⁹

Therefore, in this study, we employed ChatGPT, a state-of-the-art language model recognized for its natural language understanding capabilities, to explore the nuanced insights derived from the analysis.¹⁰ ChatGPT excels in contextual understanding, allowing comprehension of the nuanced meanings within the context of a conversation. This contextual awareness is particularly valuable in sentiment analysis, where the interpretation of sentiments often relies on an understanding of the surrounding text. In addition, ChatGPT is adaptable to diverse language nuances, capturing the intricacies of expression across different linguistic styles, cultural variations, and demographic factors.¹¹ This adaptability can be crucial in sentiment analysis, especially when dealing with social media discourse, where language can be highly dynamic and varied.

In particular, the sentiment analysis conducted in this study using ChatGPT holds several strengths. Firstly, the objective was to compare the attitudes towards vaccines identified in online content with actual vaccination rates. While analyzing data extracted from internet posts through sentiment analysis contributes to understanding a multitude of opinions, there has been a limitation in verifying how these expressed opinions relate to real-world behavioral outcomes. Analyzing the relationship with societal behavioral outcomes, as reflected in actual vaccination rates, will aid in comprehending the results of sentiment analysis and considering future analytical directions.¹² Secondly, sentiment analysis conducted in Korean, a relatively smaller cultural sphere, has encountered limitations despite various approaches. This study sought to explore the extent to which ChatGPT could contribute to understanding the subtle nuances in internet posts written by diverse individuals in the Korean language.¹³

This study aimed to conduct a sentiment analysis of Internet postings using OpenAI GPT-3.5-turbo model, exploring the potential associations between diverse opinions expressed on social media and actual vaccination rates. The goal was to investigate the causal relationship between sentiments observed through sentiment analysis and real-world behavior. We hypothesized that a higher prevalence of positive vaccine-related posts correlates with elevated vaccination rates. Additionally, the research intended to further analyze the reasons associated with attitudes toward vaccines as expressed in internet posts.

Literature review

Recent studies have provided significant insights by analyzing public sentiment towards COVID-19 vaccines on social media. These studies primarily identify positive or negative reactions to evaluate public perceptions of the vaccine.¹⁴ Sentiment analysis of tweets using natural language processing techniques has been effective in analyzing changing attitudes toward vaccines over time and across different countries. Analyzing 11 million tweets from 180 countries revealed an increasing trend in positive responses to vaccines over time.¹⁵ Furthermore, it was found that providing sufficient information is essential for fostering positive attitudes toward vaccines.¹⁶

Such data is crucial for formulating public health policies and communication strategies related to vaccines. When examining the correlation between these results and actual vaccination rates, it was reported that higher numbers of positive tweets correlate with higher vaccination rates.¹⁵ Additionally, studies have shown that regions with more positive tweets tend to have higher vaccination rates, especially among populations aged 40 and above.¹⁷ However, distrust and misinformation also contribute to lower vaccination rates.¹⁸ Therefore, providing accurate information and fostering positive public sentiment are essential to increase vaccination rates.

To achieve reliable results in these studies, effective sentiment analysis is essential. However, sentiment analysis of tweets presents several challenges, such as sarcasm, irony, and language-specific challenges.¹⁹ Current natural language processing technologies like Vader and TextBlob are mostly optimized for English data, making them difficult to use for analyzing minor languages like Korean.²⁰ Even when new Korean lexicons are developed for research, their accuracy ranges from 40-70%, which leads to inconsistent results.²¹ Therefore, advanced AI-based natural language processing tools like ChatGPT, which excel in understanding complex language patterns and nuances, show promise in achieving high accuracy across various languages and dialects.²² This capability is expected to enhance the reliability of sentiment analysis.

Methods

This observational study analyzed users’ perceptions of the COVID-19 vaccine on Internet platforms targeting working individuals and their relationship with actual vaccination rates. This study, involving the collection and analysis of publicly available internet posts without containing personal information, received approval for a consent waiver and exemption from review through the National Health Insurance Service Ilsan Hospital Institutional Review Board (IRB) (NHIMC-2023-08-028).

Results

The analysis used OpenAI’s gpt-3.5-turbo model, which yielded 851 emotional evaluations. In three cases, the model reported uncertainty; upon manual review, these instances were deemed uncertain and excluded from the analysis. Two psychiatrists evaluated 100 posts, resulting in 85% agreement with the results provided by the gpt-3.5-turbo model. The results of the confusion matrix are presented in the Extended data.

Over the 60-week analysis period, the sentiment distribution was as follows: 44 positive (5%), 704 negative (83%), and 103 neutral (12%) posts. For posts collected over 31 weeks starting from October 11, 2022, and associated with additional vaccinations during the winter season, the sentiment distribution was 20 positive (6%), 254 negative (80%), and 43 neutral (14%) posts. The total vaccination counts, weekly vaccination averages, and weekly post-vaccination averages for each study period are presented in Table 1.

Relationship between attitudes toward COVID-19 vaccination in internet postings and actual vaccination rates

Positive posts regarding vaccination attitudes showed a weak positive correlation with the first-dose vaccination counts. Negative posts on vaccines exhibited a moderately positive correlation with both first- and second-dose vaccinations. Posts expressing a neutral attitude showed a weak positive correlation with second-dose vaccinations ( Table 2, Figure 1).

Table 2. Correlation between number of vaccinations and postings by attitude toward vaccines.

	1st Dose	2nd Dose	Total	Booster vaccination
Positive attitude	0.27*	0.22	0.26*	0.50**
Negative attitude	0.59**	0.66**	0.66**	0.69**
Neutral attitude	0.24	0.31*	0.29*	0.18
Total number of postings	0.58**	0.65**	0.65**	0.69**

** The correlation is significant at the 0.01 level (2-tailed).

* The correlation is significant at the 0.05 level (2-tailed).

Figure 1. Comparison between the number of posts about vaccination (left axis) and actual vaccination (right axis) for 60 weeks.

Regarding the booster vaccination count during the winter season, a strong correlation was observed between positive and negative posts regarding vaccination attitudes.

The total counts of positive, negative, and neutral posts showed strong correlations with the counts of the first, second, and booster vaccinations. Table 2 presents the correlation results.

Extraction of attitudinal reasons toward vaccination and association analysis between reasons

Using ChatGPT, we extracted the reasons for positive or negative attitudes toward vaccines from Internet posts. Among the 44 positive posts, the most prevalent (73%) was prevention, cited in 39 instances. For the 704 negative posts, the most common reason (28%) was distrust of the social system, which was mentioned in 194 cases. Two psychiatrists manually reviewed 100 posts and compared the outcomes in the extracted results. For the first reason per post, there were 58 instances of agreement, 37 for the second reason, and 14 for the third reason.

An association analysis was also conducted for the reasons mentioned in the postings. However, this aspect was excluded from further study because of the low agreement rate (14%) for the third reason per post during the manual review. Cases in which reasons showed a high frequency of repetition were classified into item categories. The positive reasons for vaccination include prevention, symptom alleviation, reduced mortality rate, effectiveness, safety, fewer side effects, containment of the spread of infection, and immune system reinforcement. Negative reasons for vaccination included six items: mistrust of the social system, antipathy toward social oppression, side effects, concerns about side effects, lack of information, and perception of insufficient efficacy. If the first and second reasons for a post fell under the aforementioned classification, they were included in the association analysis. In the study of positive posts, 36 groups were used to analyze positive posts, and 381 groups were analyzed.

In the analysis of positive reasons, “Decreasing mortality rate” is associated with “Symptom alleviation,” showcasing a support of 5.71%, confidence of 50.00%, and a lift of 2.92. Additionally, instances of “Post-recovery symptom alleviation” (support = 5.71%, confidence = 100.00%, lift = 1.21%), “Immune activity strengthening” (support = 17.14%, confidence = 100.00%, lift = 1.21%), “Inhibition of infection spread” (support = 31.43%, confidence = 100.00%, lift = 1.21%), and “Safety” (support = 14.29%, confidence = 83.33%, lift = 1.01%) are correlated with the item of “Prevention.”

In the analysis of negative reasons, ‘Antipathy to social oppression’ strongly correlates with ‘Mistrust’ (support = 21.78%, confidence = 94.32%, lift = 2.16). Conversely, ‘Mistrust’ is moderately associated with ‘Antipathy to social oppression’ (support = 21.78%, confidence = 50.00%, lift = 2.16). Instances involving concerns about underlying conditions were linked to side effects (support = 29.13%, confidence = 91.74%, lift = 1.26). Conversely, side effects were associated with concerns about the underlying conditions (support = 29.13%, confidence = 40.07%, lift = 1.26). The presence of “lack of information” was strongly associated with”unknown side effects” (support = 23.62%, confidence = 90.91%, lift = 1.25). The overall trends in the association analyses are shown in Figure 2.

Figure 2. Association rules visualization for vaccine sentiments: positive (A) and negative (B) reasons.

Discussion

This study explored the relationship between professionals’ attitudes toward the COVID-19 vaccine as expressed in internet posts and actual vaccination rates. Additionally, it examined the positive and negative reasons associated with vaccine attitudes and investigated the relationships between them.

The sentiment analysis conducted using OpenAI’s gpt-3.5-turbo model demonstrated an 85% concordance rate when compared with evaluations by mental health specialists. This result indicates a substantial level of accuracy of the model and affirms the utility of automated analysis using natural language processing technology. Recent studies using deep-learning-based analysis models have reported sentiment analysis accuracies ranging from approximately 70% to 90%. These studies often evaluate sentiments in posts related to COVID-19 on major social media platforms, such as Twitter.^24,25 In particular, the accuracy for positive posts was significantly decreased in this study. Data imbalances in sentiment analysis could be a cause of this, leading to models overfitting. This results in proficient detection of negative sentiments—due to their prevalence in training datasets—and reduced accuracy in identifying positive sentiments.²⁶ The reasoning analysis in this study, which aimed at extracting the reasons behind positive or negative sentiments toward vaccines, showed a lower matching rate (58%). Compared to sentiment analysis, which gauges the overall sentiment, reasoning analysis delves into specific words or phrases that explain why a sentiment is expressed. This process involves a deeper understanding of language nuances and contextual cues, making it a more intricate task.²⁷ Furthermore, ChatGPT tends to include irrelevant content when producing results for short posts, making it challenging to infer the reasons. This behavior is likely attributable to the generative nature of ChatGPT, a characteristic inherent in creative AI models.¹¹

Previous survey studies have shown that negative opinions towards vaccines accounted for 39.8%,²⁸ whereas in the present study, negative opinions are markedly higher at 82%. These results are associated with certain social phenomena in which negative or extreme online content is prevalent. Cyber venting^29,30 is a phenomenon in which internet users express dissatisfaction, stress, or anger online, making it easy to express various negative emotions related to vaccination, discomfort about side effects, and social pressure. Moreover, the freedom to swiftly respond and exchange opinions through comments in anonymous online spaces can result in uninhibited emotional expressions.^31,32 This phenomenon was evident in the posts analyzed in this study, in which various expressions of anxiety and discomfort regarding vaccine side effects, derogatory remarks about compliant attitudes toward vaccination, and rumors related to vaccines were observed.

In this study, the correlation analysis between the number of posts and vaccine doses administered indicated a strong correlation between the total number of positive, negative, and neutral posts and the counts of the first, second, and additional vaccine doses. Numerous studies have explored the relevance of social media platforms to epidemiological patterns and medical information.³³ For instance, even before the COVID-19 pandemic, research suggested a strong correlation between the regional distribution of social media posts related to infectious diseases and their actual spread.³⁴ Additionally, studies on trend analysis using search queries have shown associations with the spread of infectious diseases.³⁵ In this study, the diverse opinions expressed online before and after the vaccination could be considered a reflection of the various perspectives emerging on online platforms. The strong correlation between these negative opinions and actual vaccination rates can be inferred from the content of internet posts. In Korea, the total vaccination rate reported by the Korea Disease Control and Prevention Agency was very high at 96.9%. However, the content of internet posts revealed instances where individuals were vaccinated due to social discomfort or job requirements, as well as due to social pressure. Additionally, posts expressing social discomfort due to refusal to vaccinate were also found.

In this study, implications were derived by analyzing the underlying factors influencing attitudes toward vaccines. For positive attitudes toward vaccines, individuals expected preventive effects, personal relief from symptoms and after-effects, and societal benefits, such as preventing the spread of infection. In contrast, negative attitudes toward vaccines were associated with resentment and distrust toward social oppression. This aligns with discussions in numerous studies during the pandemic, indicating that public distrust of efforts to prevent epidemics at the societal level can lead to strong resistance.^36–38 Information gaps in various media channels regarding vaccines can reinforce public anxiety and contribute to negative attitudes.³⁹ Therefore, it is crucial to implement effective communication strategies and educational initiatives to address these concerns and promote a more informed perspective on vaccination.^40,41

The limitations of this study include its focus on SNS users among employed individuals, warranting future research that encompasses diverse demographic groups and regions for a more comprehensive understanding. Furthermore, research incorporating various factors is essential to investigate the relationship between opinion formation on Internet platforms and vaccination behavior. Additionally, limited number of posts poses significant constraints on this study, compared to other studies.^6,34 This limitation may be attributed to the fact that the BLIND community, which was the focus of the analysis, primarily targets professionals, resulting in a predominance of posts related to salary, career, and other job-related topics. The period of the study, which occurred about one year after the initial vaccine rollout in February 2022, coincided with a significant decrease in public interest in vaccines, which also likely contributed to the fewer posts related to vaccines.

Moreover, the analysis of sentiment analysis models, including the gpt-3.5-turbo model, reveals additional limitations. These models may not fully capture the subtle nature of emotions, and the interpretation of emotional expressions can vary among individuals.¹¹ Therefore, involving a diverse range of reviewers across different age groups and backgrounds for manual review is crucial. Considering the gpt-3.5-turbo model, there is a limitation in fully understanding subtle nuances and cultural contexts. This is particularly evident in contexts like Korea, characterized by unique language styles, cultural norms, and demographic characteristics.¹³

Building on these limitations, this study also encountered challenges in addressing data imbalance during sentiment analysis. Traditional undersampling methods were unsuitable due to the limited dataset size, as reducing the number of negative postings could compromise the reliability of correlation analysis. Similarly, oversampling techniques such as SMOTE, which generate synthetic data in vector form, were impractical for sentiment analysis using language models like ChatGPT. To address these issues, data augmentation was performed using ChatGPT⁴² with controlled prompts to diversify positive and neutral data while maintaining contextual consistency.

Despite these efforts, the sentiment analysis results did not improve after augmentation. Positive and neutral data proportions were increased to 25-30% of the total dataset, yet the accuracy declined, with neutral data frequently misclassified as negative. This decline likely stemmed from the nondeterministic nature of ChatGPT,⁴³ leading to inconsistencies in the augmented data, and potential quality degradation caused by inaccuracies in the initial analysis, particularly in ambiguous or sarcastic cases. These findings further underscore the complexities and limitations of applying language models like ChatGPT to sentiment analysis, particularly in culturally nuanced contexts.

In conclusion, this study revealed the interaction between the public’s perception of the COVID-19 vaccine expressed on social media and their actual vaccination behavior. The perception of risk and willingness to be vaccinated can be influenced by various mass media sources, such as the news. Opinions encountered on SNS, which people use, are also likely to significantly impact individuals’ perceptions of vaccines due to biased approaches to information and the phenomenon of conformity. Therefore, implementing social strategies that provide appropriate vaccine information in an accessible manner is crucial.