Data acquisition, the key to the data life cycle, defines the data product profile. At this stage, structured and unstructured data are gathered from different sources and different types of unstructured or dirty data are pre-processed. Short data loading times are crucial for BDA due to its naturally exponential growth rate.

The most essential stage of processing Big Data is to implement a method to extract the necessary data from the loaded, un-structured Big Data. A data analyst spends the most time on cleaning dirty data. Analysing dirty data could lead to erroneous results. To get high-quality data, faulty records, duplicates, unwanted records, and outliers need to be removed. Typos must be fixed and the data requires structuring. An exploratory analysis could investigate the initial characteristics of data and helps refining the hypothesis.

The cleaned data obtained needs to be aggregated for processing numerical and categorical types of data, followed by data integration. Different types of data in various shapes and sizes obtained from different sources need to be integrated to prepare for analysis. To unify some data features, we may need to convert between formats. For example, one source collecting ratings on a five-star scale, and another source collecting data as “up” and “down” vote only. The response variable could be, y ∈ 1 2 3 4 5

and y ∈ positive negative

Before integrating both source data, we need to make an equivalent response representation, possibly by converting the first source to the second representation format, considering three stars and above as the positive ratings and the rest of them as the negative ratings. Properly integrated data becomes less complex, more centralized and more valuable.

From the perspective of Big Data, the goal is to produce meaningful insights that will be invaluable for business, through the analysis of data which may fluctuate depending on analytics technique and data types. Reports investigating the data must be constructed to help the business for better and faster decision-making.

Data interpretation allows to present data in an understandable format for users, for example, presenting data using analysis and modelling results to make decisions by interpreting the outcomes and extracting knowledge. Data interpretation queries are categorized together and indicate the same table, diagram graph or other data demonstration options.

We collected data from the UCI repository ²⁴ and from publicly available datasets in the Kaggle database. ²⁷ We stored these datasets on Kaggle’s server and worked on these data on the database’s kernels. All collected datasets were in CSV (Comma-separated values) format.

Data preprocessing is an important phase of data analysis. Raw data is manipulated to make it understandable. This is carried out in several steps, such as cleaning, encoding, imputing, among others. We handled these steps separately.

For this step, we tried to fix any errors and remove inconsistency. We fixed typos and different representations for any values in a common representation. We used fuzzy matching or edit distance algorithms to remove inconsistency. Outlier detection and removal help to get better accuracy. Figure 2 shows summary statistics (number summary) to represent data, such as, minimum, maximum, median, quartiles (Q1, Q3). The first quartile (Q1) is the middle value between the smallest value and the median (or the 50 ^th percentile, or Q2) of the dataset. A 25% portion of values in the dataset resides below the first quartile. Interquartile Range , IQR = Q 3 − Q 1 minimum = Q 1 − 1.5 ∗ IQR maximum = Q 3 + 1.5 ∗ IQR

IQR, or midspread, or middle 50%, is the statistical dispersion equal to the range from lower quartile (25th percentile) to the upper quartile (75th percentile). The values that do not reside within the range of the minimum and maximum value are defined as outliers ( Figure 2).

We prepared statistical data values in numerical and categorical values. The standard statistical types such as numeric and categorical had similar representations in Pandas ²⁹ and Python (version 3.10). To treat each feature correctly, we encoded each column as its respective type of data, which helps to apply transformation consistency in further analytical processes.

We fixed the missing values in this step. We used 0 as the default value for missing numeric data and ‘None’ as the default value for missing categorical data. We used different techniques to impute missing values and train the machine learning model by feeding these imputed datasets. Based on the model performances, we chose the best imputation technique and used it for further analytical process. The implemented algorithmic steps were as follow:

Step 1: Retrieve sample clean dataset ( Dataset clean ) from the original dataset, excluding missing/incomplete values as much as possible.

Step 2: Order Features ( Features o ) based on feature utility scores or mutual scores.

Step 3: Select top features from Features o and apply step 5.

Step 4: Select the rest of the features from Features o and apply step 6.

Step 5: For a given feature F i : label F i as the target and the rest of the column in Dataset clean as features, and train the ML model to obtain missing or incomplete values for the original dataset.

Step 6: For a given feature F i : calculate statistical parameters (mean or median) of the F i column in Dataset clean , and obtain missing or incomplete values for the original dataset.

Using mutual score is a great way to determine a feature’s potential. Feature utility scores help to determine important features and non-important ones as well. Based on scores, we discarded some features for a performance gain.

Label encoding can be done to transform categorical features, as we are focusing on the tree-ensemble model; this works for both ordered and unordered data categories. Creating new features can be done in several ways such as, taking the product of two numerical features, the square root of a feature, normalize by applying logarithms, determining the group statistics of a feature, etc.

The unsupervised algorithm k-means clustering can be used to create features as well. Cluster labels or the distance of each entity to each cluster can be used as features. Sometimes, these help to untangle complicated relationships between features, engineered features or targets.

We can use another unsupervised principal component analysis (PCA) model for feature creation, which can decompose a variational structure. The PCA algorithm gave us loadings which described each component of a variation, and the components which were the transformed datapoints. The loadings can suggest features to create and the components we can directly use as features. Clustering can be done using one or more components.

It is an encoding of categorical into numeric values derived from the target. It resembles a supervised feature engineering technique. We used mean and median values for this purpose.

A great way of boosting performance is carrying out hyperparameter tuning. For our ML model XGBoost, we set max_depth to 6, learning_rate to 0.01, n_estimators to 1000.

We adopted the K-fold cross-validation for performance evaluation. Cross-validation divided the data set into training and a testing data set to train the model and test its performance using two distinct data sets. Training and testing on the same data create overfitting issues. To avoid this, we used K-fold cross-validation with a value of 5 for K ( Figure 4). All our experimental results were five-fold cross-validated.

The XGBoost model performance was evaluated using the root mean squared logarithmic error (RMSLE) metric. The formula for RMSLE is represented as follows: RMSLE = 1 n ∑ i = 1 n log p i + 1 − log a i + 1 2

Where:

n is the number of observations in the dataset

p i is the prediction of target

a i is the actual target for i.

log( x) is the natural logarithm of x ( log e x ).

Discarding columns or rows is a technique for handling missing values. Our model performance in RMSLE was 0.14249 after discarding columns with missing values.

We evaluated our XGBoost model using datasets with imputed missing values using different types of imputation techniques. When we filled non-muerical (NAN, not a number) values with a 0, we obtained an RMSLE score of 0.14351, while when filling missing values with the next valid value on the same column, we obtained a score of 0.14348. If we use the statistical mean of a feature column to impute missing values in that column, we notice a performance increment with an RMSLE score of 0.14157.

As we carried out feature transformation and target encoding based on feature utility scores, we yielded better performances. The use of K-means clustering and PCA led to a better performance as well. We obtained a value of 0.14044 for the RMSLE score.

Hyperparameter tuning gave a performance boost in the final performance evaluation. Figure 5 shows the performance improvements after feature engineering and hyperparameter tuning. After fine-tuning some parameters, we obtained our highest RMSLE score with0.12426.

-

Ames housing dataset: house sales data in Ames, Iowa between 2006 and 2010. Compiled by Dean De Cock; used for educational purposes. We used a version of that dataset available at https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data

Analysis code available from: https://github.com/FuadAhmad/smartic

Archived analysis code at time of publication: https://zenodo.org/badge/latestdoi/420156995

License: (must be open access) Apache-2.0 License