COVID-19 Vaccine: Predicting Vaccine Types and Assessing Mortality Risk Through Ensemble Learning Algorithms

Dataset

The raw data of individuals who received vaccinations and reported adverse reactions was obtained from the VAERS.¹³ This dataset contains vaccination information for individuals vaccinated against a variety of diseases including COVID-19, Polio, Tetanus, and Influenza. However, our current study omitted any non-SARS-CoV-2 (COVID-19) vaccination information. Therefore, the dataset being used consists of 49,810 individuals. This dataset has various attributes of individuals’ information such as age, gender, current illness, medical history, allergic history, type of vaccine, life-threatening illness, symptoms after vaccinations, etc. Some of these attributes have been found to be textual (e.g., medical history, symptoms text, etc.), while others have been found to be numerical (such as age, number of doses, etc.). The description of some different attributes in the VAERS data set is illustrated in Table 1.

Preprocessing

The quality of raw data used to perform any analysis heavily influences its outcome. Therefore, the preprocessing and exploratory analysis of data becomes the most important parts of any data-driven investigation. The preprocessing of a dataset involved examining the data for missing values, irrelevant values, replicas, etc. whereas EDA assists in understanding data by visualizing it. It has been noticed that the dataset contains many missing and irrelevant values.

Any COVID-19 vaccine types that were not specified were removed, and only two types of values in the sex field were considered: “M” as male and “F” as female. Unknown values were excluded. In the died field, ‘Y’ was considered yes, and the rest were considered ‘no’; in the ‘prior vaccine’ field, ‘yes’ was considered yes, and the rest were considered ‘no’. The analysis of allergic history included considering mentioned allergic effects as positive cases and considering ‘null’, ‘none’, ‘NA’, and other negatively mentioned text as negative cases. The History column in the dataset contained written records of coexisting conditions, requiring the extraction of all of the patient's medical history separately. To better understand the patient's medical history, information about pre-existing chronic and non-chronic diseases, such as chronic obstructive pulmonary disease, hypertension, diabetes, and kidney disease, was extracted. All missing values (i.e., empty, null) were excluded from this field, and spelling/grammar mistakes were fixed.

In the Feature extraction step, most of the important features in the acquired dataset are presented as textual data. However, in order to analyze them, they must be separated into separate entities. As a result, String matching was used to convert all text data into attributes. The correlation plot (Figure 2) did not demonstrate a significant relationship between various attributes and vaccine types. Yet, previous studies revealed a direct correlation between vaccine adverse reactions and medical and allergic histories. Therefore, the number of unique entries for the diseases in the patient's medical histories was counted. Diseases with more than 300 counts in patients' medical histories were considered attributes, while the rest were ignored due to the large dataset and the computational burden associated with each individual disease. This study, therefore, considered 21 diseases which are diabetes mellitus, thyroid, different pain, obesity, migraine, kidney disease, hypertension, hyperlipidemia, high cholesterol, heart disease, Gastroesophageal Reflux Disease (GERD), depression, dementia, positive history of COVID-19, Chronic Obstructive Pulmonary Disease (COPD), cancer, atrial fibrillation, asthma, arthritis, anxiety, and anemia from the patient’s medical history as attributes. Using the VAERS id, these files have been merged into one file after identifying and extracting features. The analyzed dataset has 28 different features and over 49,810 samples. The data was encoded using a one-hot encoding technique.

Figure 2. Correlation plot between different features of the VAERSA dataset.

Data-Sampling Algorithms

In this study, only three methods of handling imbalanced data are used. In the first place, no changes are made to the data. Normally, it is divided into training and testing data at a ratio of 8 to 2. This first technique is referred to as “Normal” in this study. Next, experiments are conducted using well-known imbalanced data techniques called SMOTE, Tomek-links, and SMOTETOMEK, for balancing the dataset which combines SMOTE and Tomek links.¹⁴ As with the previous experiment, the dataset is divided into training and testing data at a ratio of 8 to 2. This experiment aims to handle imbalanced data and further improve the performance of machine learning classification models, especially in the multiclass classification scenario. This experiment aims to handle imbalanced data and further improve the performance of machine learning classification models, especially in the multiclass classification scenario. These Sampling methods were selected considering data nature, imbalance ratio, algorithm compatibility, and analysis goals.

Description of Ensemble Methods

To predict which vaccine will be most effective for a candidate without causing severe adverse reactions (output) based on several factors (input), different machine-learning algorithms were used to build the proposed model. These approaches were selected due to their accuracy, robustness, efficiency, scalability, and ability to handle large, high-dimensional datasets while reducing overfitting.

Random Forest (RF)

A multipurpose data mining approach for classification. It is based on decision trees that operate as an ensemble, an approach of combining multiple classifiers to identify problems and enhance accuracy. A classification is predicted by each tree independently, and votes for the relevant class, and the majority of votes decide the model’s prediction. It can handle large dataset with high dimensionality, it also improves the accuracy of the model and eliminates the overfitting problem.¹⁵

Decision Tree (DT)

A DT is a supervised learning technique that can be used for classification and regression problems; however, it is most commonly used to resolve classification issues. In this tree-organized classifier, the internal nodes represent datasets, branches represent decision rules, and each leaf node represents the outcome. A DT has two nodes: the decision node and the leaf node. The leaf nodes are the result of such decisions and they do not have any extra branches, but decision nodes are frequently used to settle any decision and have several branches. Based on the features of the dataset, decisions or tests are made.¹⁶

Extreme Gradient Boosting (XGB)

XGBoost is an ensemble learning method combining multiple weak models' predictions to generate a stronger prediction. In the beginning, XGB fits the data to a weak classifier. Afterward, the data is fitted to another weak classifier to increase accuracy without affecting the current model. In the same way, the process continues until the best accuracy is achieved.¹⁷ Furthermore, XGBoost supports parallel processing, making it possible to train models on large datasets in a reasonable period of time.

Light Grading Boosting Machine (LGBM)

LGBM is an open-source gradient boosting algorithm based on a tree-based learning framework; it is an open-source GBDT algorithm designed by Microsoft Research Asia. This framework grew trees vertically (leaf-wise) rather than horizontally (level-wise) as other tree-based frameworks did. Therefore, it can reduce the losses more efficiently and handle huge dataset with less computational complexity due to its lighter version.¹⁸

Model performance evaluation

Macro average and Weighted Average are used to calculate the performance of the four classifiers used for learning.

In general, a confusion matrices are a 2 × 2 matrix. Where rows represent the instances in the actual class, and the columns represented the predicted class. It results in four possible outcomes: TP, FP, TN, and FN.

Figure 3. Confusion matrix examples.

(a) Binary classification confusion matrix. (b) Multiclass classification confusion matrix.¹⁹

Using the above outcomes, we can check whether the predictions are correct.^16,20,21

ROC Curves show the performance of the classification model across all classification thresholds. In a ROC curve, the TP rate and FP rate are plotted at each threshold of classification. “AUC” stands for “Area Under the ROC Curve”. It can be used as a classifier to distinguish between classes. In general, the higher the AUC value, the better the classifier is at identifying positive from negative classes.^22,23

Algorithms Hyperparameter Tuning Using Grid Search

Hyperparameter tuning is a crucial step in the data mining model development process, involving the refinement of hyperparameters within a data mining algorithm to uncover the optimal combination that enhances classifier performance. The Grid Search approach is a widely recognized and effective method for hyperparameter tuning.²⁴ In the context of this paper, Grid Search was employed using the GridSearchCV object from scikit-learn to thoroughly explore and identify the hyperparameter set that consistently produces the most favorable results. Known for its systematic and methodical approach to hyperparameter tuning, Grid Search operates by specifying a set of hyperparameters and their potential values, creating a grid of all possible combinations, and assessing the model's performance for each. This method exhaustively searches through the grid, identifying the hyperparameters that consistently yield the best results, and fine-tunes the model for optimal performance.

A comprehensive overview of the default parameters for the data mining classifiers is provided in the appendix. Additionally, it details the parameters specifically assigned to each classifier for the purpose of randomized parameter optimization to enhance performance.

Multiclass classification results: based upon both medical history and vaccine type

This section presents the results of the multiclass classification for covid-19 vaccine predicting problem, along with the analysis and the discussion. Firstly, we considered the patient’s medical history as independent features and the vaccine type (value 0 means Moderna, 1 means Pfizer, and 2 means Janssen) as dependent features that depend on the independent features. Then each of the three data-sampling procedures—SMOTE, TOMEK-LINKS, and SMOTETOMEK—was applied separately. Figure 6 illustrates the effects of applying various data-balancing techniques.

The performance parameters for each model on the test dataset are presented in Table 2. As a result, the following observations have been noted:

ROC curves have been used to further analyze the predictive capability of these developed models, which are shown in Figure 8. The RF and DT models prove their effectiveness. Taking AUC into account, all developed models perform satisfactorily.

Figure 8. ROC curves for covid-19 type multiclass classification.

Binary classification results

In our model’s analysis, firstly, we considered the patient’s medical history as the independent features, and the vaccine type (value 0 means Moderna and value 1 means Pfizer) and the patient death (value 0 mean alive, and value 1 mean died) as dependent features. We trained and evaluated our models using test data by measuring accuracy, precision, recall, and AUC.

Scenario 1: Based upon both medical history and vaccine type

The performance parameters for each model on the test dataset are presented in Table 3. As a result, the following observations have been noted:

ROC curves have been used to further analyze the predictive capability of these models, which are shown in Figure 9. The RF and DT models prove their effectiveness. Taking AUC into account, all developed models perform satisfactorily.

Figure 9. Scenario 1: ROC Curve for binary vaccine type dataset.

Scenario 2: based upon both medical history and death

The patient’s death dataset was also experimented with as the vaccine-type dataset. Figure 10 demonstrates the effect of applying various data-sampling methods. The performance parameters for each model on the test dataset are presented in Table 4. As a result, the following observations have been noted:

Figure 10. Class label counts before and after applying the various data-sampling techniques for the death dataset.

ROC curves have been used to further analyze the predictive capability of these models, which are shown in Figure 11. The RF and DT models prove their effectiveness. Taking AUC into account, all developed models perform satisfactorily.

Figure 11. Scenario 2: ROC curve for a Death.

The importance of all the features in the COVID-19 vaccine adverse reactions dataset is calculated using the feature importance package from the Scikit-learn Python library. A visual representation of the calculated values for feature importance is displayed in Figure 12. The features are arranged based on their respective importance scores.

Figure 12. Ranking of features based on the patients' medical history coefficient values.

Figure 12 shows that patients' age, gender, and use of other medicines were significant factors in the past medical history of all target variables. WHEN examining the target variable of “vaccine type,” the analysis revealed a comprehensive set of critical attributes within the patient's medical history that strongly influence the selection of the administered vaccine. These attributes include previous vaccine history, allergic history, diabetes, arthritis, hypertension, and asthma. Furthermore, when investigating the target variable of death status, certain factors emerged as highly significant. These factors include heart disease, allergic history, dementia, hypertension, diabetes, kidney disease, and Chronic obstructive pulmonary disease (COPD). These attributes have shown a noteworthy impact on the desired outcome, indicating their importance in predicting the death status of patients.

The patient's age and gender provide essential demographic information that may impact the choice of vaccine, as certain vaccines have age or gender-specific recommendations. Additionally, considering the patient's current medication usage is crucial to ensure compatibility and potential interactions with the chosen vaccine. Previous vaccine history helps determine if the patient requires a booster or a specific type of vaccine.

The presence of underlying conditions such as diabetes, arthritis, allergic history, hypertension, and asthma is highly influential in the decision-making process. These conditions may affect the patient's immune response or make them more susceptible to certain vaccine side effects. By considering these attributes, healthcare professionals can tailor the vaccine type to maximize efficacy and minimize risks for each patient.

Comparing Proposal Models with Related Works

This section illustrates the comparison of our proposal model with the results of prediction models that are available in the related works. The comparison was structured around the methodologies employed and the achieved levels of accuracy. Table 5 presents the findings from four distinct studies on COVID-19 vaccine side effects. These studies utilized data from Twitter and the VAERS (COVID-19 World Vaccine Adverse Reactions dataset) spanning across different years. Each study employs a unique set of techniques to achieve specific objectives, resulting in varying degrees of accuracy.

Strengths and limitations

As far as the authors are aware, this is the first study that attempts to predict the type of covid-19 vaccine appropriate for a candidate, along with the death probability risk. Additionally, we suggest approaches to address the issue of imbalanced data concerning adverse reactions to COVID-19 vaccines.

This study has some limitations. Because these data were collected online, we cannot rule out information-gathering bias in the study. Moreover, this data set contained a significant amount of missing data, which may lead to a misrepresentation of patient populations.

Conclusion

In this work, four ML models were evaluated: DT, RF, XGBoost, and LGBM. Three sampling techniques were executed for each model to handle imbalanced data. Below are some of the key findings of the study, which shed light on crucial insights and implications:

According to the study's results, the RF model is recommended for machine learning tasks that demand high accuracy and robustness. While both the XGBoost and LGBM models are also viable options, the RF model could be preferable when dealing with imbalanced data. The effectiveness of these balancing algorithms has been evaluated, leading to the conclusion that no single technique can consistently produce the best results across all datasets. When considering the importance of data distribution, machine learning techniques and balancing algorithms are both crucial.

Future Works

The findings of this study can be extrapolated to various other datasets related to vaccinations. While the inclusion of medical history features was restricted due to the substantial size of the dataset and the computational complexities associated with processing each disease, there is room for further advancement. By automating the system, its capability to analyze predictions based on a broader spectrum of medical history features can be enhanced. As new data streams into the dataset, fresh predictions can be dynamically generated by this automation, considering the prevailing factors at that specific moment. Additionally, the integration of deep learning methodologies presents an opportunity to uncover latent patterns within the data, thereby enhancing comprehension of the intricate dynamics governing COVID-19 vaccine acceptability. This multifaceted approach is poised not only to augment predictive accuracy but also to deepen the understanding of the nuanced interplay between medical history, vaccination patterns, and evolving epidemiological dynamics.