exprso: an R-package for the rapid implementation of machine learning algorithms

Machine learning plays a major role in many scientific investigations. However, non-expert programmers may struggle to implement the elaborate pipelines necessary to build highly accurate and generalizable models. We introduce exprso, a new R package that is an intuitive machine learning suite designed specifically for non-expert programmers. Built initially for the classification of high-dimensional data, exprso uses an object-oriented framework to encapsulate a number of common analytical methods into a series of interchangeable modules. This includes modules for feature selection, classification, high-throughput parameter grid-searching, elaborate cross-validation schemes (e.g., Monte Carlo and nested cross-validation), ensemble classification, and prediction. In addition, exprso also supports multi-class classification (through the 1-vs-all generalization of binary classifiers) and the prediction of continuous outcomes.

Supervised machine learning has an increasingly important role in biological studies. However, the sheer complexity of machine learning pipelines poses a significant barrier to expert biologists unfamiliar with the intricacies of machine learning. Moreover, many biologists lack the time or technical skills necessary to establish their own pipelines. Here, we discuss the exprso package, a framework for the rapid implementation of highthroughput machine learning, tailored specifically for use with high-dimensional data. As such, this package aims to empower investigators to execute state-of-the-art binary and multi-class classification, as well as regression, with minimal programming experience necessary.
Although R offers a tremendous number of high-quality machine learning packages, there exists only a handful of fully integrated machine learning suites for R. Of these, we recognize here the caret package which offers an expansive toolkit for both classification and regression analyses 1 . Otherwise, we acknowledge the RWeka package which provides an API to the popular Weka machine learning suite, originally written in Java 2 . While these packages have a vast repertoire of functionality, we believe the exprso package has some advantages.
First, this package employs an object-oriented design that makes the software intuitive to lay programmers. In place of a few, elaborate functions that offer power at the expense of convenience, this package makes use of more, simpler functions whereby each constituent event has its own method that users can combine in tandem to create their own custom analytical pipeline. Second, this package contains single functions that execute elaborate high-throughput machine learning pipelines. These, coupled with special argument handlers, manage sophisticated pipelines such as high-throughput parameter gridsearching, Monte Carlo cross-validation 3 , and nested crossvalidation 4 . Moreover, users can embed these high-throughput modules (e.g., parameter grid-searching) within other modules (e.g., Monte Carlo cross-validation), allowing for infinite possibility. In addition, this package provides an automated way to build ensembles from the results of these high-throughput modules.
In addition, this package facilitates multi-class classification by generalizing binary classification methods to a multi-class context. Specifically, this package automatically executes 1-vs-all classification and prediction whenever working with a dataset that contains multiple class labels. Moreover, this package provides a specialized high-throughput module for 1-vs-all classification with individual 1-vs-all feature selection, an alternative to conventional multiclass classification that has been reported to improve results (at least in the setting of 1-vs-1 multi-class support vector machines) 5 . The exprso package also supports the prediction of continuous outcomes.
While we acknowledge that premier machine learning suites, like caret, may surpass our package in the breadth of their functionality, we do not intend to replace these tools. Rather, we developed exprso as an adjunct, or alternative, tailored specifically to those with limited programming experience, especially biologists working with high-dimensional data. That said, we hope that even some expert programmers find value in our software.

Implementation
This package uses an object-oriented framework for machine learning. In this paradigm, every unique task, such as data splitting (i.e., creating the training and validation sets), feature selection, and model construction, has its own associated function, called a method. These methods typically work as wrappers for other R functions, structured so that the objects returned by one method will feed seamlessly into the next method.
In other words, each method represents one of a number of analytical modules that provides the user with stackable and interchangeable data processing tools. Examples of these methods include wrappers for popular feature selection methods (e.g., analysis of variance (ANOVA), recursive feature elimination 6,7 , empiric Bayes statistic 8 , minimum redundancy maximum relevancy (mRMR) 9 , and more) as well as numerous models (e.g., support vector machines (SVM) 10 , neural networks 11 , deep neural networks 12 , random forests 13 , and more).
We have adopted a nomenclature to help organize the methods available in this package. In this scheme, most functions have a few letters in the beginning of their name to designate their general utility. Below, we include a brief description of these function prefixes along with a flow diagram of the available methods.
• array: Modules that import data stored as a data.frame, ExpressionSet object, or local text file. Alternatively, the exprso function imports data in x, y format (recommended for most users).
• mod: Modules that modify the imported data prior to building models.
• split: Modules that split these data into training and validation (or test) sets.
• fs: Modules that perform feature selection.
• build: Modules that build models and ensembles.
• predict: Modules that deploy models and ensembles.
• calc: Modules that calculate model performance, including area under the receiver operating characteristic (ROC) curve (AUC). • pl: Modules that manage pipelines, including highthroughput parameter grid-searches, Monte Carlo cross-validation, and nested cross-validation.
• pipe: Modules that filter the pipeline results.
We refer the reader to the package vignette, "An Introduction to the exprso Package," hosted with the package on the Comprehensive R Archive Network (CRAN), for a detailed description of the functions available from this package 16 .

Operation
Specific computer hardware requirements will depend on the size of the dataset under study and the functions used. For the most part, however, a standard laptop computer with the latest version of R installed will handle most applications of the exprso package.

Use cases
To showcase this package, we make use of the publicly available hallmark Golub 1999 dataset to differentiate (i.e., classify) acute lymphocytic leukemia (ALL) from acute myelogenous leukemia (AML) based on gene expression as measured by microarray technology 17 . We begin by importing this dataset as an ExpressionSet object from the package GolubEsets (version 1.16.0) 18 . Then, using the arrayExprs function, we load the ExpressionSet object into exprso. Note that, alternatively, one could use the exprso function to import the data in x, y format (recommended for most users).
Next, using the modFilter, modTransform, and modNormalize methods, we threshold filter, log2 transform, and standardize the data, respectively, reproducing the pre-processing steps taken by the original investigators 19 . To keep the code clear and concise, we make use of the %>% function notation from the magrittr package 20 . In short, this function passes the result from the previous function call to the first argument of the next function, known as piping.
library(exprso) library(golubEsets) library(magrittr) data(Golub_Merge) array <-arrayExprs(Golub_Merge, colBy = "ALL.AML", include = list("ALL","AML"))%>% modFilter(20, 16000, 500, 5) %>% modTransform %>% modNormalize Then, using the splitSample method, one of the split methods shown in the above diagram, we partition the data into a training and a test set through random sampling without replacement. Next, we perform a series of feature selection methods on the extracted training set. Through the fs modules fsStats and fsPrcomp, we pass the top 50 features as selected by the Student's t-test through dimension reduction by principal components analysis (PCA).  When constructing a model using a build module, we can only specify one set of parameters at a time. However, investigators often want to test models across a vast range of parameters. For this reason, we provide methods like plGrid to automate high-throughput parameter grid-searches. These methods not only wrap model construction, but also model deployment. In addition, they accept a fold argument to toggle leave-one-out or v-fold cross-validation.
Below, we show a simple example of parameter grid-searching, whereby the top 3, 5, and 10 principal components, as established above, get used to construct linear and radial kernel SVMs with costs of 1, 101, and 1001. In addition, we calculate a (biased) 10-fold cross-validation accuracy to help guide our choice of the final model parameters. (Note that we call this accuracy biased because we are performing cross-validation on a dataset that has already undergone feature selection. Although this approach gives a poor assessment of absolute classifier performance 23 , it may still have value in helping to guide parameter selection in a statistically valid manner. As an alternative to this biased cross-validation accuracy, users can instead call the plNested method in which feature selection is performed anew with each data split that occurs during the leave-one-out or v-fold cross-validation.) Finally, we show an example for the plMonteCarlo method, an implementation of Monte Carlo cross-validation. Compared to the plGrid method which iteratively builds and deploys models on a validation (or test) set, plMonteCarlo wraps multiple iterations of data splitting, feature selection, and parameter grid-searching. The final result therefore contains a summary of the model performances as measured across any number of bootstraps carved out from the initial dataset. Argument handler functions help organize the arguments supplied to the splitting, feature selection, and high-throughput methods of the plMonteCarlo method call. (Note that when using the Monte Carlo cross-validation method (or any of the other pl modules), the user may iterate over any build method provided by exprso, not only buildSVM. This includes the buildDNN method for deep neural networks as implemented via h2o 12 . Also note that the user can embed other cross-validation methods, such as another Monte Carlo or nested method, within the cross-validation method call, allowing for endless combinatory possibility.) In the first section of the code below, we define the argument handler functions for the plMonteCarlo call. As suggested by their names, the ctrlSplitSet, ctrlFeatureSelect, and ctrlGridSearch handlers manage arguments to data splitting, feature selection, and high-throughput grid-searching, respectively. In this example, we set up arguments to split the unaltered training set through random sampling with replacement, perform the two-step feature selection process from above, and run a high-throughput parameter grid-search with biased crossvalidation. The unaltered dataset is processed this way 10 times, as directed by argument B.
ens <-buildEnsemble(boot, colBy = "valid. acc", top = 1) pred <-predict(ens, testSet(splitSets)) The above procedure, including building and deploying ensembles, also works for multi-class classification and continuous outcome prediction. We refer the reader to the package vignettes, "An Introduction to the exprso Package" and "Advanced Topics for the exprso Package", both hosted with the package on the Comprehensive R Archive Network (CRAN), for a detailed description of all methods included in this package 16 .

Summary
Here we introduce the R package exprso, a machine learning suite tailored specifically to working with high-dimensional data. Unlike other machine learning suites, we have prioritized simplicity of use over expansiveness. As such, exprso provides a fully interchangeable and modular programming interface that allows for the rapid implementation of classification and regression pipelines. We have included in this framework functions for executing some of most popular feature selection methods and machine learning algorithms. In addition, exprso also contains a number of modules that perform high-throughput parameter grid-searching in conjunction with sophisticated cross-validation schemes. Owing to its ease-of-use and extensive documentation, we hope exprso will serve as a helpful resource, especially to scientific investigators with limited prior programming experience.

Software availability
Software available from: http://cran.r-project.org/web/packages/ exprso/ Author contributions TQ designed and implemented the tool, applied the tool to the use case, and drafted the article. DT and SG helped design the tool and drafted the article. DT contributed code and performed extensive beta testing.

Competing interests
No competing interests were disclosed.

Grant information
The author(s) declared that no grants were involved in supporting this work.

Particular comments:
The -function should accept also standard matrix containing expression values, the arrayExps groups may then be given as character vector. Handling of most functions can be improved by providing standard parameter values, e.g: in (may simply include all groups) arrayExps percent.include in (66%) splitSampe and so on Help the users to find errors in their function calls by giving appropriate warning/error messages. E.g. "fsStats cannot be applied in multi-class studies". No non-programmer will know what an "inherited method" is. Key changes include: * Easier data import with the `exprso` function which imports data in x, y format * Software now supports continuous outcome prediction (when importing data via `exprso`) * More default values (e.g., split modules) to simplify user experience * Created custom errors for every split, feature selection, build function. Most other functions now have custom errors as well * Documentation is split up by unique function. ?mod, ?split, ?fs, ?build, etc. all open a table of contents that overviews the unique functions available * Figure 1 simplified with data sources clearly labeled in black * Manuscript adjusted to reflect other changes Also, you should not encounter any error with mutli-class classification when importing data using the new `exprso` function.
(PS: I am aware of a trivial warning in this version that is triggered by predict. It is already fixed in the developmental branch on GitHub).

Cheers, Thom
No competing interests were disclosed. Competing Interests: 09  This is a short software tool article presenting a new R package for implementing machine learning pipelines. According to the authors it is targeted specifically at non-expert programmers analyzing high-dimensional biological data. The article briefly describes design goals and implementation details of the package, and then provides an example of building full machine learning pipeline for well-known microarray data set.
The main contribution of this work lies in the prepared software and not in the accompanying manuscript. Because of this in the article there are no clear hypotheses nor conclusions. The software package does not provide any novel algorithms or functionalities. It is conceived as a wrapper which should make existing methods more accessible. This goal is of practical rather than scientific nature. Ultimately, the usefulness of the package can only be confirmed by its wider adoption by the users. This situation makes it hard to write a conclusive review.
I agree with the motivation behind this software. R package infrastructure can be indeed confusing and it is notoriously hard to navigate through it for a beginner programmer. I find the interface adopted by exprso package relatively clean and unambiguous. The way the evaluation methods are implemented is consistent with best machine learning practices. Eventual success of this package depends on how well it will be integrated with other popular packages. I hope that the authors will have the resources for further development of exprso.
Below I present more detailed comments for the authors: caret is mentioned as the R package with similar goals to exprso. I would find a more detailed comparison between these two package useful, both in terms of general design and specific use cases.
It is commendable when newly created packages integrate with existing infrastructure. As I understand, exprso has some limited integration with GEOquery and Biobase packages allowing easy data import. However, after data is loaded all operations use special ExprsArray objects, distinct from native R types such as DataFrame or Matrix. I understand this design choice, but I 1 2 1 2 distinct from native R types such as DataFrame or Matrix. I understand this design choice, but I have to note that it limits interoperability. Should the additional processing in the middle of exprso pipeline be required, data needs to be converted between formats.
I find the design in which transformations applied on the training set are automatically applied on the testing set controversial. It obscures the pipeline and may not be intuitive for beginner programmers. Moreover, sometimes transformations of the testing set differ slightly from the transformations of the training set.
It is not obvious to me why simple train-test split is implemented as "split" module and cross-validation as "pl" module. There are not very much different. The way cross-validation is currently implemented does not allow detailed control over individual folds, which is sometimes useful.
No competing interests were disclosed.

Competing Interests:
We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com