MSclassifier: median-supplement model-based classification tool for automated knowledge discovery

Implementation

The package implements median-supplement approaches to machine learning, robust machine learning techniques that have the advantage of supporting complete compliance efforts by not missing sensitive sub-datasets or allowing certain sub-datasets to escape the classification process when balancing overall datasets. They are applicable to datasets with unequal numbers of instances associated with each class (group).

Median-supplement machine learning algorithms. They involve the following steps:

There are two kinds of median-supplement approaches, namely, median-supplement Random Forest and median-supplement Naive Bayes methods. Each approach is distinguished by the kind of model constructed from the median-supplement data. For a ’median-supplement Random Forest’, a Random Forest classifier is inferred from the median-supplement data to assign classes to instances. To obtain a ’median-supplement Naive Bayes classifier’, a Naive Bayes model is developed from the median-supplement data to classify instances. The overview of the underlying principles of median-supplement approaches as implemented in MSclassifier is shown in Figure 1.

Figure 1. Overview of median-supplement models implementation in MSclassifier.

A set of medians of attributes and a randomly generated matrix with uniformly distributed values are initially derived from the training sample. The result of a scalar multiplication of medians and corresponding column vectors of the random matrix is obtained and aggregated to the initial training sample to form a median-supplemented dataset. Finally, median-supplemented models are inferred from the median-supplemented data to predict new instances.

Naive Bayes model. This model applies the Bayesian framework to predict classes of new instances. Any classes having the highest posterior probability becomes the class of a new test instance. Let G be a set of attributes. Then, the probability that any instance belongs to any class/category, C_j, is given by:

where P(G|C_j) is the probability of G given class C_j, P(C_j) is the probability of C_j and P(G) is the probability of G occurring. In this model, the attributes of each class are presumed to be independent distributions if the class is known. Thus, for each i-th attribute of n attributes, g_i, the probability is given by:

Random Forest model. Random Forest is advancement in multistage decision making. It is a collection of Decision Trees. This typically involves constructing a collection of trees from bootstrap samples each of which consists of a subset of variables of the training sets. This approach of inferring trees from bootstrap samples involves recursively repeating the following²⁰:

Once all desired trees have been achieved in those steps (which repeats after reaching a putative node size), a classification is determined by a majority vote. Assume C_b(x) is the class prediction of the b-th random forest tree. Then the classifier is given by:

Typically,$m=\sqrt{n.}$ Using random forest spans from the fact that it improves predictive accuracies of tree-based methods^19,20.

Operation of MSclassifier

MSclassifier, implemented in R, can be installed and run on most operating systems. The sole requirement is the availability of a recent version of R (https://cran.r-project.org/). The package is organized as a programme with the flexibility of selecting a median-supplement Random Forest or a median-supplement Naive Bayes method. The overview of the package follows the structure presented in Figure 1. The Documentation of the package has detailed instructions for installation and usage as well as other descriptions of the package.

MSclassifier does not require any special programming skills of the user. It accepts a tabular dataset in which the attributes and instances are in columns and rows respectively. In this way, the class of each instance is stored in the last column. At any time, two different datasets, training and test sets, may be supplied and the programme returns the predicted classes of instances of the test set. The training set comprises of characterized (labelled) samples whereas the test set is not characterized. In the absence of a test set, the user can specify only the training set to obtain a model for further analysis. Furthermore, the user specifies the desired median-supplement method. If no method is specified, a median-supplement Random Forest is automatically applied. Summary descriptions of arguments of MSclassifier function is described in Table 1. Samples of training and test sets are provided with the package. They are used in the illustration of the MSclassifier in the next section.

Illustration (usage) of MSclassifier

In order to illustrate the use of the package, we use HER2 datasets included in the package. These datasets were obtained from an earlier study that explored the use of machine learning techniques to determine hormone and receptor status phenotypes in breast cancer¹⁷. The training data consists of 86 HER2 receptor-negative and 14 HER2 receptor-positive samples while the independent test set consists of 51 HER2 receptor-negative samples and 11 HER2 receptor-positive samples. The illustration shows how to use the MSclassifier after installation.

Comprehensive description of function and application can be found in the help file after loading the installed package and getting the full description of the package:
> library(MSclassifier)  		# load package
> ?MSclassifier

To load package datasets:

> data(her2)
> data(testset)

To view a subset of the package’s her2 training set:
> her2[1:3,1:3]
      NPTXR_23467 DOCK3_1795 LOC400927_400927
1    266.0075         38.1356          12.7119
2    461.8575         34.9231           6.5481
3    199.3335         11.8146           6.9676

To view a subset of the package’s her2 test set:

> testset[1:3,1:3]
      NPTXR_23467  DOCK3_1795  LOC400927_400927
1    304.5058          21.0756           0.7267
2    453.4778          10.1620           7.5072
3    510.8080            3.5776           4.9292

To classify instances/determine her2 status of the test samples using median-supplement Random Forest, the following apply:

> Predictions <- MSclassifier(her2,testset = testset, method ="MSRandomForest")
> head(Predictions)
Sample1  Sample2  Sample3  Sample4  Sample5  Sample6
Negative Negative Positive Negative Negative Negative
Levels: Negative Positive

To analyse median-supplement Random Forest for error matrix, here is a sample:

 > Model <- MSclassifier(her2, testset = NULL, method = "MSRandomForest")
 > predictions <- predict(Model, newdata = testset)
 > head(predictions)
 1        2        3        4        5        6
 Negative Negative Positive Negative Negative Negative
 Levels: Negative Positive

 > table(predictions, testset$her2_status)
  predictions Negative Positive
  Negative       47          10
  Positive        4           1

 To classify instances/determine her2 status of the test samples using median-supplement Naive Bayes, here is a sample:

 > Predictions <- MSclassifier(her2,testset = testset, method = "MSNaiveBayes")
 > head(Predictions)
 Sample1  Sample2  Sample3  Sample4  Sample5  Sample6
 Negative Negative Positive Negative Negative Negative
 Levels: Negative Positive

To analyse median-supplement Naive Bayes for error (confusion) matrix, the following is an example:

> Model <- MSclassifier(her2, testset = NULL, method = "MSNaiveBayes")
> predictions <- predict(Model, newdata = testset[,-ncol(testset)])
> head(predictions)
[1] Negative Negative Positive Negative Negative Negative
Levels: Negative Positive

> table(predictions, testset$her2_status)
predictions Negative Positive
Negative       51          8
Positive        0          3

Data sets. In order to illustrate the performance of the package, we use two real datasets. Particularly, the first data, obtained from previous study¹⁷ describes gene expression measurements in breast cancer. In this illustration, median-supplement models are inferred with the MSclassifier package to assign classes to new instances of the test set. In the case of the HER2 data, the class of each instance is the expressed receptor status phenotype while attributes are the relevant gene expression profiles. The data consists of 86 HER2 receptor-negative and 14 HER2 receptor-positive samples. These are samples included in the MSclassifier package.

The second (larger) dataset was derived from a study that characterized amino acid sequences of human proteins localized in nine cellular compartments²¹. Code written in LISP was used to determine values of physicochemical properties of proteins known to be primarily localized in the designated subcellular locations were used. Protein properties used are based on the amino acid composition (including hydrophobicity, normalized van der Waals volume, polarity, polarizability, and charge), transitions and distribution as detailed²¹. For instance, "PERCENT-R" is a reference to the percentage of arginine residues in the primary sequence of amino acids of a protein; "HYDROPHOBICITY-PERCENT-GROUP1" is a reference to the percentage of polar amino acids in the primary sequence of amino acids (i.e. group 1 amino acids are polar, group 2 amino acids are neutral, and group 3 amino acids are hydrophobic); "POLARITY-GP1-GP3-TRANSITIONS" is a reference to the frequency of transitions between low polarity residues (L, I, F, W, C, M, V, and Y) and high polarity residues in a given protein’s primary sequence of amino acids (H, Q, R, K, N, E, D). The data comprised of 2635 instances and 126 attributes. Among the instances, 1589 were associated with (localized in) the plasma-membrane and 1046 were associated with the nucleus²². In its usage to illustrate the package, instances of the dataset were classified as "nucleus" and "plasma-membrane".

Performance measures of packages. The performance of each method is determined by its classification rate: proportion of correctly classifying instance given by the ratio of correctly classified test instances to the total number of test instances²³. In general, the classification rates agree with measures of accuracies of such classification methods. Higher classification rate of a method indicates that the package has higher chances of making accurate assignments of samples to their respective classes. Therefore, it is desirable to have higher classification rate. For instance, a higher classification rate for classifying receptor status phenotypes in breast cancer indicates the method has high sensitivity for deciphering the particular receptor status. This is because the sensitivity is also a proportion of correctly classified instances among characterized instances as exemplified in unsupervised learning systems²⁴. Furthermore, Mann-Whitney tests are performed to evaluate differences among classification rates of the methods. Both independent and cross-validation testing methods are used to evaluate the packages²². While a 10-fold cross-validation is applied to the HER2 data, a 5-fold cross-validation is applied to the subcellular localization of proteins data²².

Performance on independent test sample

In this experiment, HER2 training and test sets made available in the package were used. It was found that the median-supplement Naive Bayes (MNB) implemented in MSclassifier outperformed all the other methods considered in this case (Figure 2). This was to be expected since the MSclassifier implements median-supplement methods, which have been shown to outperform the traditional machine learning methods¹⁷. Higher performance of this package on this test example is the result of the enhanced median-supplement training set from which MSclassifier infers models. Thus the enhancement makes more instances available to train models.

Figure 2. Comparison of MSclassifier and other packages in R.

NB is Naive Bayes, RF is Random Forest, MRF is median-supplement Random Forest and MNB median-supplement Naive Bayes. While both MRF and MNB are implemented in MSclassifier, NB is implemented in e1071 and RF is implemented in randomForest packages.

Performances from cross-validation testing

The classification rates of conventional methods, implemented in existing packages, ranged between 83% and 91%, methods implemented in the MSclassifier had values with minimum of 87% and maximum of 91%. Particularly, it was found that conventional random forest was significantly higher than the Naive Bayes (mean classification rate of 89.83% versus 85.43%, p = 1.48e-11). However, the median-supplement Naive Bayes implemented in MSclassifier achieved the highest classification rates among all the methods²². More importantly, it had significantly higher classifications rates than the random forest method (mean is 90.30% versus 89.83%, p = 8.62e-3). These results are consistent with performance of median-supplement methods on HER2 classifications studied earlier¹⁷.

With regards to the prediction of subcellular localization of proteins, although both MSclassifier and the other packages could attain equally high classification rates (94%) in this test, the minimum classification rate achieved by the median-supplement Naive Bayes was lower compared to the conventional Naive Bayes method (mean of 69% versus 86%, p = 4.55e-14). However, this observation was different in other studies¹⁷. The difference is attributable to the differences in data and prediction tasks. Nevertheless, these performances are suboptimal when compared to the random forest-based methods which achieved mean classification rates of 93%²². Specifically, the performances of both the random forest and the median-supplement random forest were statistically indistinguishable (mean of 93.42% versus 93.19%, p = 0.06). These results are indicative that tree-based random forest methods have better performances on larger datasets. However, the superiority of median-supplement methods over several other machine learning methods when applied to predict hormone and HER2 receptor phenotypes underpinned in the literature¹⁷. These results demonstrate the potential of MSclassifier to better predict instances of binary classifications problems.

Underlying data

Datasets used in Use Case:

Extended data

Zenodo: Supporting information and data for MSclassifier: Median-Supplement model-based Classification tool for automated knowledge discovery, https://doi.org/10.5281/zenodo.3946675²².

This project contains the following extended data: