Predicting Outcomes of Hormone and Chemotherapy in the Molecular&nbsp;Taxonomy of&nbsp;Breast Cancer&nbsp;International&nbsp;Consortium (METABRIC) Study by Biochemically-inspired Machine Learning

Iman Rezaeian; Eliseos J. Mucaki; Katherina Baranova; Huy Q. Pham; Dimo Angelov; Alioune Ngom; Luis Rueda; Peter K. Rogan

doi:10.12688/f1000research.9417.1

Home Browse Predicting Outcomes of Hormone and Chemotherapy in the MolecularTaxonomy...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Note

Predicting Outcomes of Hormone and Chemotherapy in the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) Study by Biochemically-inspired Machine Learning

[version 1; peer review: 2 approved with reservations]

Iman Rezaeian¹, Eliseos J. Mucaki², Katherina Baranova², [...] Huy Q. Pham¹, Dimo Angelov³, Alioune Ngom¹, Luis Rueda¹, Peter K. Rogan ^2-4

Iman Rezaeian¹, Eliseos J. Mucaki², [...] Katherina Baranova², Huy Q. Pham¹, Dimo Angelov³, Alioune Ngom¹, Luis Rueda¹, Peter K. Rogan ^2-4

PUBLISHED 31 Aug 2016

Author details Author details

¹ Department of Computer Science, University of Windsor, Windsor, Canada
² Department of Biochemistry, University of Western Ontario, London, Canada
³ Department of Computer Science, University of Western Ontario, London, Canada
⁴ CytoGnomix Inc, London, Canada

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Bioinformatics gateway.

This article is included in the Machine learning: life sciences collection.

Abstract

Genomic aberrations and gene expression-defined subtypes in the large METABRIC patient cohort have been used to stratify and predict survival. The present study used normalized gene expression signatures of paclitaxel drug response to predict outcome for different survival times in METABRIC patients receiving hormone (HT) and, in some cases, chemotherapy (CT) agents. This machine learning method, which distinguishes sensitivity vs. resistance in breast cancer cell lines and validates predictions in patients, was also used to derive gene signatures of other HT (tamoxifen) and CT agents (methotrexate, epirubicin, doxorubicin, and 5-fluorouracil) used in METABRIC. Paclitaxel gene signatures exhibited the best performance, however the other agents also predicted survival with acceptable accuracies. A support vector machine (SVM) model of paclitaxel response containing the ABCB1, ABCB11, ABCC1, ABCC10, BAD, BBC3, BCL2, BCL2L1, BMF, CYP2C8, CYP3A4, MAP2, MAP4, MAPT, NR1I2, SLCO1B3, TUBB1, TUBB4A, TUBB4B genes was 78.6% accurate in 84 patients treated with both HT and CT (median survival ≥ 4.4 yr). Accuracy was lower (73.4%) in 304 untreated patients. The performance of other machine learning approaches were also evaluated at different survival thresholds. Minimum redundancy maximum relevance feature selection of a paclitaxel-based SVM classifier based on expression of ABCB11, ABCC1, BAD, BBC3 and BCL2L1 was 79% accurate in 53 CT patients. A random forest (RF) classifier produced a gene signature (ABCB11, ABCC1, BAD, BCL2, CYP2C8, CYP3A4, MAP4, MAPT, NR1I2, TUBB1, GBP1, OPRK1) that predicted >3 year survival with 82.4% accuracy in 420 HT patients. A similar RF gene signature showed 79.6% accuracy in 504 patients treated with CT and/or HT. These results suggest that tumor gene expression signatures refined by machine learning techniques can be useful for predicting survival after drug therapies.

Keywords

Gene expression signatures, breast cancer, chemotherapy resistance, hormone therapy, machine learning, support vector machine, random forest

Corresponding author: Peter K. Rogan

Competing interests: PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests.

Grant information: AN and LR are funded by NSERC grants RGPIN-2016-05017 and RGPIN-2014-05084 and by the Windsor Essex County Cancer Centre Foundation under a Seeds4Hope grant. PKR has been supported by NSERC [Discovery Grant 371758-2009], Canadian Foundation for Innovation, Canada Research Chairs and Cytognomix Inc.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2016 Rezaeian I et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

How to cite: Rezaeian I, Mucaki EJ, Baranova K et al. Predicting Outcomes of Hormone and Chemotherapy in the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) Study by Biochemically-inspired Machine Learning [version 1; peer review: 2 approved with reservations]. F1000Research 2016, 5:2124 (https://doi.org/10.12688/f1000research.9417.1) First published: 31 Aug 2016, 5:2124 (https://doi.org/10.12688/f1000research.9417.1) Latest published: 12 May 2017, 5:2124 (https://doi.org/10.12688/f1000research.9417.3)

Introduction

Current pharmacogenetic analysis of chemotherapy makes qualitative decisions about drug efficacy in patients (determination of good, intermediate or poor metabolizer phenotypes) based on variants present in genes involved in the transport, biotransformation, or disposition of a drug. We have applied a supervised ML approach to derive accurate gene signatures, based on the biochemically-guided response to chemotherapies with breast cancer cell lines¹, which show variable responses to growth inhibition by paclitaxel and gemcitabine therapies^2,3. We analyzed stable⁴ and linked unstable genes in pathways that determine their disposition. This involved investigating the correspondence between 50% growth inhibitory concentrations (GI₅₀) of paclitaxel and gemcitabine and gene copy number, mutation, and expression first in breast cancer cell lines and then in patients¹. Genes encoding direct targets of these drugs, metabolizing enzymes, transporters, and those previously associated with chemo-resistance to paclitaxel (n=31 genes) were then pruned by multiple factor analysis (MFA), which indicated expression of ABCC10, BCL2, BCL2L1, BIRC5, BMF, FGF2, FN1, MAP4, MAPT, NKFB2, SLCO1B3, TLR6, TMEM243, TWIST1, and CSAG2 could predict sensitivity in breast cancer cell lines with 84% accuracy. The cell line-based paclitaxel-gene signature predicted sensitivity in 84% of patients with no or minimal residual disease (n=56; data from 5). The present study derives related gene signatures with ML approaches that predict outcome of hormone- and chemotherapies in the large METABRIC breast cancer cohort⁶.

Methods

SVM learning: Previously, paclitaxel-related response genes were identified from peer-reviewed literature, and their expression and copy number in breast cancer cell lines were analyzed by multiple factor analysis of GI₅₀ values of these lines² (Figure 1). Genes with expression levels related to GI₅₀ were used to derive SVMs by backwards feature selection for paclitaxel, tamoxifen, methotrexate, 5-fluorouracil, epirubicin, and doxorubicin (trained using the function fitcsvm in MATLAB R2014a⁷ and tested with either leave-one-out or 9 fold cross-validation). These SVMs were then assessed for their ability to predict patient outcomes based on available metadata (see Figure 1 and reference 1). Interactive prediction using normalized expression values as input is available at http://chemotherapy.cytognomix.com.

Figure 1. Biochemically-inspired SVM gene signature derivation workflow.

The initial set of genes is carefully selected through the understanding of the drug and the pathways associated with it. A multiple factor analysis of the GI₅₀ values of a training set of breast cancer cell lines and the corresponding expression levels of each gene in the initial set reduces the list of genes. Given this expression levels of each gene the reduced set for each cell line, the method finds the optimal gene subset and the SVM that minimizes the misclassification rate by cross-validation. The SVM is evaluated on patients by classifying those with shorter survival time as resistant and longer survival as sensitive to hormone and/or chemotherapy. The Gaussian kernel SVM requires manual selection of two different parameters, C and sigma; these parameters determine how strictly the SVM learns the training set, thus if not selected properly can lead to overfitting. A grid search evaluates a wide range of combinations of these values by parallelization. The algorithm selects the C and sigma combination that lead to the lowest cross-validation misclassification rate. A backwards feature selection (greedy) algorithm is used, in which one gene of the set is left out in a reduced gene set and the classification is then assessed; genes that maintain or lower the misclassification rate are kept in the signature. The procedure is repeated until the subset with the lowest misclassification rate is selected as the optimal subset of genes.

RF learning: RF was trained using the WEKA 3.7⁸ data mining tool. This classifier uses multiple random trees for classification, which are combined via a voting scheme to make a decision on the given input gene set. Figure 2 depicts the therapy outcome prediction process of a given patient using a RF consisting of a series of decision trees derived from different subsets of paclitaxel-related genes.

Figure 2. RF decision tree diagram depicts the therapy outcome prediction process of a given patient, using a RF consisting of k decision trees.

Several DTs are built using different subsets of paclitaxel-related genes. The process starts from the root of each tree and if the expression of the gene corresponding to that node is greater than a specific value, the process continues through the right branch, otherwise it continues through the left branch until it reaches a leaf node; that leaf represents the prediction of the tree for that specific input. The decisions of all trees are considered and the one with the largest number of votes is selected as the patient outcome.

Augmented Gene Selection: The most relevant genes (features) for therapy outcome prediction were found using the minimum redundancy and maximum relevance (mRMR) approach⁹. mRMR is a wrapper that incrementally selects genes by maximizing the average mutual information between gene expression features and classes, while minimizing their redundancies:

m R M R = max_{s} [\frac{1}{| s |} \sum_{f_{i} \in S} I (f_{i}, C) - \frac{1}{{| s |}^{2}} \sum_{f_{i}, f_{j} \in S} I (f_{i}, f_{j})]

where f_i corresponds to a feature in gene set S, I(f_i,C) is the mutual information between f_i and class C, and I(f_i,f_j) is the mutual information between features f_i and f_j.

For this experiment, we used a 26-gene signature (genes ABCB1, ABCB11, ABCC1, ABCC10, BAD, BBC3, BCL2, BCL2L1, BMF, CYP2C8, CYP3A4, MAP2, MAP4, MAPT, NR1I2, SLCO1B3, TUBB1, TUBB4A, TUBB4B, FGF2, FN1, GBP1, NFKB2, OPRK1, TLR6, TWIST1) as the base feature set. These genes were selected (in Ref. 1) based either on their known involvement in paclitaxel metabolism, or evidence that their expression levels and/or copy numbers correlate with paclitaxel GI₅₀ values (Table 3). mRMR and SVM were combined to obtain a subset of genes that can accurately predict patient survival outcome; here, we considered 3, 4 and 5 years as survival thresholds for breast cancer patients (Table 3).

Results and discussion

Supplemental Dataset: Predicted Treatment Response for Each Individual METABRIC Patient
				Predicted Response with SVM Analysis Methods (described in Table 1)										Predicted Response with Random Forest (Described in Table 2)																		Predicted Response with application of mRMR and SVM (Described in Table 3)
METABRIC_ID	Living Status	Treatment (CT - Chemotherapy HT - Hormone RT - Radiation)	Time Since Treatment (Years)	Observed Response Median Survival Threshold of 4.4 years (Sens./Resist. - 0/1)	CT and HT Only Both Datasets (Paclitaxel)	CT and HT Only Both Datasets (Tamoxifen)	CT and HT Only Both Datasets (Methotrexate)	CT and HT Only Both Datasets (Epirubicin)	CT and HT Only Both Datasets (Doxorubicin)	CT and HT Only Both Datasets (5-Fluorouracil)	CT and/or HT 'Discovery' Dataset	Deceased (CT and/or HT) 'Discovery' Dataset	No Treatment Both Datasets	CT Observed Response >3 year	CT Predicted Response >3 year	CT Observed Response >4 year	CT Predicted Response >4 year	CT Observed Response >5 year	CT Predicted Response >5 year	HT Observed Response >3 year	HT Predicted Response >3 year	HT Observed Response >4 year	HT Predicted Response >4 year	HT Observed Response >5 year	HT Predicted Response >5 year	CT and/or HT Observed Response >3 year	CT and/or HT Predicted Response >3 year	CT and/or HT Observed Response >4 year	CT and/or HT Predicted Response >4 year	CT and/or HT Observed Response >5 year	CT and/or HT Predicted Response >5 year	CT Observed Response >3 year	CT Predicted Response >3 year	CT Observed Response >4 year	CT Predicted Response >4 year	CT Observed Response >5 year	CT Predicted Response >5 year	HT Observed Response >3 year	HT Predicted Response >3 year	HT Observed Response >4 year	HT Predicted Response >4 year	HT Observed Response >5 year	HT Predicted Response >5 year	CT and/or HT Observed Response >3 year	CT and/or HT Predicted Response >3 year	CT and/or HT Observed Response >4 year	CT and/or HT Predicted Response >4 year	CT and/or HT Observed Response >5 year	CT and/or HT Predicted Response >5 year
MB-0220	d	CT	3.27	1	1	1	1	0	0	1	0	1	-	0	0	1	1	1	1	-	-	-	-	-	-	0	0	1	0	1	0	0	0	1	1	1	1	-	-	-	-	-	-	0	0	1	0	1	1
MB-0259	d	CT	0.97	1	1	1	1	1	1	1	-	-	-	1	0	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	1	1	1	-	-	-	-	-	-	1	1	1	0	1	0
MB-0272	d	CT	10.02	1	1	0	0	0	0	1	0	0	-	0	1	0	1	0	1	-	-	-	-	-	-	0	0	0	0	0	1	0	0	0	1	0	1	-	-	-	-	-	-	0	1	0	0	0	0
MB-0278	d	CT	0.48	1	1	1	1	1	1	1	-	-	-	1	0	1	0	1	1	-	-	-	-	-	-	1	1	1	0	1	0	1	0	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-0333	d	CT	2.18	1	1	1	1	0	1	1	-	-	-	1	0	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-0346	d	CT	1.68	1	1	1	1	1	1	1	1	1	-	1	0	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	1	1	1	-	-	-	-	-	-	1	0	1	1	1	0
MB-0354	d	CT	0.95	1	1	0	1	1	1	1	0	0	-	1	0	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	0	1	1	1	1	-	-	-	-	-	-	1	1	1	1	1	1
MB-0361	d	CT	1.24	1	1	1	0	1	0	1	1	1	-	1	0	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	1	1	0	1	1	1	0	-	-	-	-	-	-	1	0	1	0	1	0
MB-0400	d	CT	1.85	1	1	1	1	1	1	1	0	1	-	1	1	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-0446	a	CT	4.84	0	0	0	1	1	0	1	1	-	-	0	1	0	1	1	1	-	-	-	-	-	-	0	0	0	0	1	0	0	0	0	0	1	1	-	-	-	-	-	-	0	0	0	0	1	0
MB-0467	a	CT	3.79	0	0	0	0	1	1	0	1	-	-	0	0	1	1	1	1	-	-	-	-	-	-	0	0	1	0	1	0	0	0	1	1	1	0	-	-	-	-	-	-	0	0	1	0	1	0
MB-0564	d	CT	3.12	1	1	1	0	1	1	1	1	1	-	0	0	1	1	1	1	-	-	-	-	-	-	0	1	1	0	1	1	0	0	1	1	1	0	-	-	-	-	-	-	0	0	1	0	1	1
MB-0663	a	CT	4.55	0	0	1	0	0	0	1	-	-	-	0	0	0	0	1	1	-	-	-	-	-	-	0	1	0	0	1	0	0	0	0	0	1	0	-	-	-	-	-	-	0	0	0	0	1	0
MB-2614	d-d.s.	CT	5.33	1	1	1	1	1	1	1	0	0	-	0	0	0	1	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	1	0	1	-	-	-	-	-	-	0	0	0	0	0	1
MB-2718	a	CT	13.30	0	0	1	1	0	1	1	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0
MB-2758	d-d.s.	CT	2.87	1	1	1	0	1	1	1	-	-	-	1	0	1	0	1	0	-	-	-	-	-	-	1	0	1	0	1	0	1	0	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-4601	d-d.s.	CT	4.62	1	1	0	1	0	1	0	-	-	-	0	0	0	0	1	0	-	-	-	-	-	-	0	0	0	0	1	0	0	0	0	0	1	1	-	-	-	-	-	-	0	0	0	0	1	0
MB-4621	a	CT	11.66	0	1	1	1	0	1	0	-	-	-	0	0	0	1	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	1	0	1	0	0	-	-	-	-	-	-	0	1	0	0	0	0
MB-4715	d-d.s.	CT	2.89	1	1	1	0	1	1	1	1	1	-	1	1	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-4731	d-d.s.	CT	3.69	1	1	1	1	0	1	1	-	-	-	0	0	1	0	1	1	-	-	-	-	-	-	0	0	1	0	1	0	0	0	1	1	1	1	-	-	-	-	-	-	0	1	1	0	1	0
MB-4746	a	CT	18.32	0	1	0	0	0	0	0	-	-	-	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-4757	d-d.s.	CT	2.25	1	0	1	1	1	1	1	1	1	-	1	0	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-4886	d-d.s.	CT	3.96	1	1	1	1	1	1	1	1	1	-	0	1	1	1	1	1	-	-	-	-	-	-	0	0	1	0	1	0	0	1	1	1	1	1	-	-	-	-	-	-	0	1	1	0	1	1
MB-4893	a	CT	14.51	0	0	1	0	1	0	0	-	-	-	0	1	0	1	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-4945	d-d.s.	CT	1.65	1	1	1	0	1	1	1	1	0	-	1	1	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-5070	a	CT	15.69	0	1	1	0	0	0	0	1	-	-	0	0	0	1	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0
MB-5072	d-d.s.	CT	4.13	1	1	1	1	0	0	0	0	1	-	0	1	0	1	1	1	-	-	-	-	-	-	0	0	0	0	1	0	0	0	0	1	1	1	-	-	-	-	-	-	0	0	0	0	1	0
MB-5229	d-d.s.	CT	2.79	1	1	1	1	1	1	1	0	1	-	1	0	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	0	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-5298	d-d.s.	CT	6.97	1	1	1	1	1	1	1	-	-	-	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	1	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-5312	d-d.s.	CT	3.11	1	1	1	0	1	1	0	1	1	-	0	1	1	1	1	1	-	-	-	-	-	-	0	0	1	0	1	1	0	1	1	1	1	1	-	-	-	-	-	-	0	0	1	0	1	0
MB-5348	a	CT	13.94	0	0	1	0	1	1	0	0	-	-	0	0	0	1	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	1	0	0	0	1	-	-	-	-	-	-	0	1	0	0	0	0
MB-5411	d-d.s.	CT	2.14	1	1	1	1	1	1	1	1	1	-	1	0	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	1	1	1	-	-	-	-	-	-	1	1	1	0	1	0
MB-5458	a	CT	13.39	0	1	0	0	0	1	0	-	-	-	0	1	0	1	0	0	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	1	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-5474	a	CT	13.93	0	0	0	0	0	0	0	1	-	-	0	1	0	1	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0
MB-5558	a	CT	2.06	0	1	0	1	1	1	0	-	-	-	1	1	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	1	1	0	-	-	-	-	-	-	1	0	1	0	1	0
MB-5625	d-d.s.	CT	2.65	1	0	0	0	1	1	0	1	1	-	1	0	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-6058	d-d.s.	CT	1.86	1	1	1	1	1	1	1	-	-	-	1	0	1	1	1	1	-	-	-	-	-	-	1	0	1	0	1	0	1	0	1	1	1	1	-	-	-	-	-	-	1	1	1	0	1	0
MB-6059	a	CT	2.35	0	1	0	0	0	0	1	-	-	-	1	0	1	1	1	1	-	-	-	-	-	-	1	0	1	1	1	0	1	1	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-6063	d-d.s.	CT	1.28	1	1	0	1	1	1	0	-	-	-	1	0	1	0	1	0	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	0	1	1	-	-	-	-	-	-	1	0	1	0	1	0
MB-6082	d-d.s.	CT	8.42	1	1	1	1	0	1	0	-	-	-	0	0	0	1	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	1	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-6098	d-d.s.	CT	0.87	1	1	1	1	0	1	0	-	-	-	1	0	1	1	1	0	-	-	-	-	-	-	1	0	1	0	1	0	1	1	1	1	1	0	-	-	-	-	-	-	1	1	1	0	1	0
MB-6113	d-d.s.	CT	10.23	1	1	0	1	1	1	1	-	-	-	0	0	0	0	0	1	-	-	-	-	-	-	0	1	0	1	0	1	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-6114	a	CT	5.49	0	0	1	1	0	1	0	-	-	-	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-6122	a	CT	4.39	0	0	0	0	0	1	1	-	-	-	0	0	0	0	1	0	-	-	-	-	-	-	0	0	0	0	1	0	0	0	0	0	1	1	-	-	-	-	-	-	0	1	0	0	1	0
MB-6160	a	CT	10.27	0	0	0	1	0	1	0	-	-	-	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-6246	d	CT	4.34	1	1	1	1	1	1	1	-	-	-	0	0	0	0	1	0	-	-	-	-	-	-	0	0	0	0	1	0	0	0	0	0	1	1	-	-	-	-	-	-	0	0	0	0	1	0
MB-6273	a	CT	8.62	0	1	0	1	1	0	0	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-7020	d-d.s.	CT	5.87	1	1	1	1	1	1	1	-	-	-	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-7073	a	CT	9.52	0	0	0	1	0	0	0	-	-	-	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-7112	d-d.s.	CT	5.31	1	1	1	1	1	0	0	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-7158	d-d.s.	CT	3.81	1	1	1	1	0	0	1	-	-	-	0	0	1	0	1	1	-	-	-	-	-	-	0	0	1	0	1	0	0	1	1	1	1	1	-	-	-	-	-	-	0	0	1	0	1	0
MB-7270	a	CT	14.39	0	0	0	0	1	1	1	-	-	-	0	0	0	0	0	1	-	-	-	-	-	-	0	0	0	0	0	0	0	0	0	0	0	0	-	-	-	-	-	-	0	0	0	0	0	0
MB-7275	d-d.s.	CT	3.65	1	1	1	1	1	1	1	-	-	-	0	0	1	0	1	1	-	-	-	-	-	-	0	0	1	0	1	1	0	1	1	1	1	1	-	-	-	-	-	-	0	0	1	0	1	0
MB-0005	a	CT/HT	8.36	0	0	0	0	1	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0
MB-0158	a	CT/HT	4.73	0	0	1	1	1	0	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	1	0	0	1	0
MB-0288	d-d.s.	CT/HT	3.24	1	1	1	1	1	1	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	0	1	0
MB-0291	d	CT/HT	3.22	1	1	1	1	1	0	1	1	1	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	0	1	0
MB-0305	d	CT/HT	5.22	1	0	1	1	1	1	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	1	0	0	0	0
MB-0321	a	CT/HT	5.26	0	0	0	0	0	0	1	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	1
MB-0335	a	CT/HT	9.86	0	0	0	1	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	1	0	0	0	0
MB-0345	a	CT/HT	0.55	0	0	0	0	1	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	1	1	1	0	1	1
MB-0392	d-d.s.	CT/HT	2.71	1	1	1	1	1	1	1	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	1	1	1	1	1	1
MB-0407	a	CT/HT	5.80	0	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0
MB-0434	d-d.s.	CT/HT	3.74	1	0	1	1	1	1	1	1	1	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	1	1	1
MB-0489	a	CT/HT	5.35	0	1	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0
MB-0524	a	CT/HT	6.64	0	1	0	0	1	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0
MB-0529	d	CT/HT	3.99	1	1	1	1	1	1	1	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	1	1	1
MB-0559	a	CT/HT	9.86	0	0	0	0	0	0	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0
MB-0609	a	CT/HT	0.99	0	0	1	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	1	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	0
MB-0614	a	CT/HT	0.70	0	0	0	0	0	1	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	1	1	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	1
MB-0899	a	CT/HT	4.53	0	0	0	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	1	0	0	1	0
MB-4935	a	CT/HT	15.95	0	0	1	1	1	0	1	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	1
MB-5299	d-d.s.	CT/HT	1.37	1	0	0	0	1	0	1	1	1	-	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	0
MB-6085	a	CT/HT	2.00	0	0	0	1	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	0
MB-6184	a	CT/HT	13.13	0	1	1	1	0	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0
MB-6217	a	CT/HT	12.29	0	1	1	0	1	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0
MB-7054	d-d.s.	CT/HT	2.50	1	1	1	0	1	1	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	1	0	1	0	1	0
MB-7057	d-d.s.	CT/HT	3.60	1	1	1	1	1	1	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	1	1	0	1	0
MB-7093	a	CT/HT	8.39	0	1	0	1	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0
MB-7097	d-d.s.	CT/HT	5.35	1	1	1	1	1	1	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	0	0
MB-7163	d-d.s.	CT/HT	4.32	1	0	0	1	1	1	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	1	0
MB-7182	d-d.s.	CT/HT	4.06	1	1	1	1	1	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	1	0
MB-7194	d-d.s.	CT/HT	3.83	1	1	1	1	1	1	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	1	0	1	0
MB-7226	d-d.s.	CT/HT	4.38	1	1	1	1	1	0	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	1	0	-	-	-	-	-	-	-	-	-	-	-	-	0	0	0	0	1	0
MB-0006	a	CT/HT/RT	4.71	0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-

Dataset 1.Predicted treatment response for each individual METABRIC patient.

The predicted and expected response to treatment for each individual METABRIC patient for each analyses listed in Table 1, Table 2 and Table 3 are indexed. Patients sensitive to treatment are labeled with ‘0’ while resistant patients are labeled ‘1’.

Table 1. SVM gene expression signature performance on METABRIC patients.

Patient treatment	# of patients	Agent: final gene signature	Accuracy (%)
CT and CT/HT combination (without radiation therapy)¹	84	Paclitaxel: ABCC1, ABCC10, BAD, BIRC5, FN1, GBP1, MAPT, SLCO1B3, TMEM243, TUBB3, TUBB4B	78.6
		Tamoxifen: ABCC2, ALB, CCNA2, E2F7, FLAD1, FMO1, NCOA2, NR1I2, PIAS4, SULT1E1	76.2
		Methotrexate: ABCC2, ABCG2, CDK2, DHFRL1	71.3
		Epirubicin: ABCB1, CDA, CYP1B1, ERBB3, ERCC1, MTHFR, PON1, SEMA4D, TFDP2	72.6
		Doxorubicin: ABCC2, ABCD3, CBR1, FTH1, GPX1, NCF4, RAC2, TXNRD1	75.0
		5-Fluorouracil: ABCB1, ABCC3, MTHFR, TP53	71.4
CT and/or HT^1,2,3,4	735	Paclitaxel: BAD, BCAP29, BCL2, BMF, CNGA3, CYP2C8, CYP3A4, FGF2, FN1, NFKB2, NR1I2, OPRK1, SLCO1B3, TLR6, TUBB1, TUBB3, TUBB4A, TUBB4B, TWIST1	66.1
Deceased only^2,4,5 (CT and/or HT)	327	Paclitaxel: ABCB11, BAD, BBC3, BCL2, BCL2L1, BIRC5, CYP2C8, FGF2, FN1, GBP1, MAPT, NFKB2, OPRK1, SLCO1B3, TMEM243	75.2
No treatment¹	304	Paclitaxel: ABCB1, ABCB11, BBC3, BCL2L1, BMF, CYP3A4, FGF2, GBP1, MAP4, MAPT, NR1I2, OPRK1, SLCO1B3, TUBB4A, TUBB4B, TWIST2	73.4

Initial gene sets preceding feature selection: Paclitaxel - ABCB1, ABCB11, ABCC1, ABCC10, BAD, BBC3, BCAP29, BCL2, BCL2L1, BIRC5, BMF, CNGA3, CYP2C8, CYP3A4, FGF2, FN1, GBP1, MAP2, MAP4, MAPT, NFKB2, NR1I2, OPRK1, SLCO1B3, TLR6, TUBB1, TWIST1. Tamoxifen - ABCB1, ABCC2, ALB, C10ORF11, CCNA2, CYP3A4, E2F7, F5, FLAD1, FMO1, IGF1, IGFBP3, IRS2, NCOA2, NR1H4, NR1I2, PIAS4, PPARA, PROC, RXRA, SMARCD3, SULT1B1, SULT1E1, SULT2A1. Methotrexate - ABCB1, ABCC2, ABCG2, CDK18, CDK2, CDK6, CDK8, CENPA, DHFRL1. Epirubicin - ABCB1, CDA, CYP1B1, ERBB3, ERCC1, GSTP1, MTHFR, NOS3, ODC1, PON1, RAD50, SEMA4D, TFDP2. Doxorubicin - ABCB1, ABCC2, ABCD3, AKR1B1, AKR1C1, CBR1, CYBA, FTH1, FTL, GPX1, MT2A, NCF4, RAC2, SLC22A16, TXNRD1. 5-Fluorouracil - ABCB1, ABCC3, CFLAR, IL6, MTHFR, TP53, UCK2.

¹ Surviving patients; ² Analysis included patients in the METABRIC ‘discovery’ dataset only; ³ SVMs tested with 9 fold cross-validation, all others tested with leave-one-out cross-validation; ⁴ Includes all patients treated with HT,CT, combination CT/HT, either with or without combination radiotherapy; ⁵ Median time after treatment until death (> 4.4 years) was used to distinguish favorable outcome, ie. sensitivity to therapy.

Table 2. Results of applying RF to predict outcome of paclitaxel therapy.

Type of treatment	Survival years (as threshold)	# Patients	Final gene expression signature	Accuracy (%)	AUC¹
Chemotherapy (CT)	3	53	ABCB1, ABCB11, ABCC1, ABCC10, BAD, BBC3, BCL2, BCL2L1, BMF, CYP2C8, CYP3A4, MAP2, MAP4, MAPT, NR1I2, SLCO1B3, TUBB1, TUBB4A, TUBB4B	56.6	0.534
	4			60.4	0.645
	5			58.5	0.645
Hormone therapy (HT)	3	420		82.4	0.641
	4			77.4	0.555
	5			70.7	0.648
CT and/or HT	3	504		79.6	0.564
	4			73.6	0.527
	5			63.5	0.563

¹AUC: Area under receiver operating curve; both Discovery and Validation patient datasets analyzed

Table 3. Results of mRMR feature selection for an SVM for predicting outcome of paclitaxel therapy.

Type of treatment	Survival years (as threshold)	Number of patients¹	Gene signature selected by mRMR	Accuracy (%)	AUC
Chemotherapy (CT)	3	53	BCL2 , TWIST1	73.6	0.724
	4		ABCB11, ABCC1, BAD, BBC3, BCL2L1	79.2	0.793
	5		ABCB11, BAD, CYP2C8, CYP3A4, MAP2, MAPT, FGF2	77.4	0.759
Hormone therapy (HT)	3	420	ABCB11, ABCC1, BAD, BCL2, CYP2C8, CYP3A4, MAP4, MAPT, NR1I2, TUBB1, GBP1, OPRK1	84.0	0.532
	4		BBC3, MAP4, FGF2, OPRK1	79.3	0.519
	5		ABCC10, MAPT, TUBB1	72.6	0.520
CT and/or HT	3	504	ABCB11, ABCC1, BBC3, BCL2, BCL2L1, CYP2C8, CYP3A4, MAP2, MAPT, TLR6, TWIST1	74.6	0.565
	4		ABCB11, BAD, BCL2, CYP3A4, MAP2, MAP4, NR1I2, OPRK1, TWIST1	75.0	0.535
	5		TWIST1, BCL2, BMF, CYP2C8, CYP3A4, BCL2L1, BBC3, TLR6, BAD, MAP4, NR1I2, GBP1, NFKB2	65.9	0.525

¹Predicted treatment responses for individual METABRIC patients using the described ML techniques are provided in Dataset 1.

The performance of several ML techniques have been compared that distinguish paclitaxel sensitivity and resistance in METABRIC patients using its tumour gene expression datasets. SVMs have generated gene signatures, indicating which genes are important for treatment response in METABRIC patients. These models are more accurate for prediction of outcomes in patients receiving HT and/or CT compared to other patient groups.

SVMs and RF were trained using expression of genes associated with paclitaxel response, mechanism of action and stable genes in the biological pathways of these targets (Figure 3). SVM models for drugs used to treat these patients were derived by backwards feature selection on patient subsets stratified by treatment or outcome (Table 1). The highest SVM accuracy was found for the paclitaxel signature in patients treated with HT and/or adjuvant chemotherapy (78.6%).

Figure 3.

Schematic elements of gene expression changes associated with response to paclitaxel. Red boxes indicate genes with a positive correlation between gene expression or copy number, and resistance using multiple factor analysis. Blue demonstrates a negative correlation. Genes outlined in dark grey are those in a previously published paclitaxel SVM model (reproduced from reference 1 with permission).

The RF classifier was used to predict paclitaxel therapy outcome for patients that underwent CT and/or HT (Table 2). The best performance achieved with RF showed 82.4% overall accuracy using a 3-year survival threshold for distinguishing therapeutic resistance vs. sensitivity.

The best overall accuracy and AUC (sensitivity and specificity) for CT/HT patients using mRMR feature selection for SVM predicting outcome of paclitaxel therapy was obtained for CT patients with 4 year survival. Outcomes for HT patients with 3 year survival were predicted with 84% accuracy; however the specificity was lower in this group. SVM combined with mRMR further improved accuracy of feature selection and prediction of response to hormone and/or chemotherapy based on survival time than either SVM or RF alone.

While not a replication study sensu stricto, the initial paclitaxel gene set used for feature selection was the same as in our previous study¹. Predictions for the METABRIC patient cohort, which was independent of the previous validation set⁵, of the either same (SVM) or different ML methods (RF and SVM with mRMR) exhibited comparable or better accuracies than our previous gene signature¹.

These techniques are powerful tools which can be used to identify genes that may be involved in drug resistance, as well as predict patient survival after treatment. Future efforts to expand these models to other drugs may assist in suggesting preferred treatments in specific patients, with the potential impact of improving efficacy and reducing duration of therapy.

Data availability

Patient data: The METABRIC datasets are accessible from the European Genome-Phenome Archive (EGA) using the accession number EGAS00000000083 (https://www.ebi.ac.uk/ega/studies/EGAS00000000083). Normalized patient expression data for the discovery (EGAD00010000210) and validation sets (EGAD00010000211) were retrieved with permission from EGA. Corresponding clinical data was obtained from the literature⁶. While not individually curated, HT patients were treated with tamoxifen and/or aromatase inhibitors, while CT patients were most commonly treated with cyclophosphamide-methotrexate-fluorouracil (CMF), epirubicin-CMF, or doxorubicin-cyclophosphamide.

F1000Research: Dataset 1. Predicted treatment response for each individual METABRIC patient, 10.5256/f1000research.9417.d133983¹⁰

Author contributions

PKR, AN and LR designed the methodology and oversaw the project. SVM feature selection with MATLAB was automated by DA. EJM and KB selected the initial gene signatures, and performed processing of the METABRIC data using SVM methods. IR performed the preprocessing of the METABRIC dataset using RF; IR and HQ designed feature selection and classification modules using WEKA. PKR, IR, EJM, AN, and LR wrote the manuscript.

Competing interests

PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests.

Grant information

AN and LR are funded by NSERC grants RGPIN-2016-05017 and RGPIN-2014-05084 and by the Windsor Essex County Cancer Centre Foundation under a Seeds4Hope grant. PKR has been supported by NSERC [Discovery Grant 371758-2009], Canadian Foundation for Innovation, Canada Research Chairs and Cytognomix Inc.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Faculty Opinions recommended

References

1. Dorman SN, Baranova K, Knoll JH, et al.: Genomic signatures for paclitaxel and gemcitabine resistance in breast cancer derived by machine learning. Mol Oncol. 2016; 10(1): 85–100. PubMed Abstract | Publisher Full Text
2. Daemen A, Griffith OL, Heiser LM, et al.: Modeling precision treatment of breast cancer. Genome Biol. 2013; 14(10): R110. PubMed Abstract | Publisher Full Text | Free Full Text
3. Shoemaker RH: The NCI60 human tumour cell line anticancer drug screen. Nat Rev Cancer. 2006; 6(10): 813–823. PubMed Abstract | Publisher Full Text
4. Park NI, Rogan PK, Tarnowski HE, et al.: Structural and genic characterization of stable genomic regions in breast cancer: Relevance to chemotherapy. Mol Oncol. 2012; 6(3): 347–59. PubMed Abstract | Publisher Full Text
5. Hatzis C, Pusztai L, Valero V, et al.: A genomic predictor of response and survival following taxane-anthracycline chemotherapy for invasive breast cancer. JAMA. 2011; 305(18): 1873–1881. PubMed Abstract | Publisher Full Text
6. Curtis C, Shah SP, Chin SF, et al.: The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature. 2012; 486(7403): 346–352. PubMed Abstract | Publisher Full Text | Free Full Text
7. MATLAB and Statistics Toolbox Release 2014a. The MathWorks Inc., Natick, Massachusetts, United States.
8. Hall M, Frank E, Holmes G, et al.: The WEKA data mining software: an update. ACM SIGKDD explorations newsletter. 2009; 11(1): 10–18. Publisher Full Text
9. Ding C, Peng H: Minimum redundancy feature selection from microarray gene expression data. J Bioinform Comput Biol. 2005; 3(2): 185–205. PubMed Abstract | Publisher Full Text
10. Rezaeian I, Mucaki EJ, Baranova K, et al.: Dataset 1 in: Predicting Outcomes of Hormone and Chemotherapy in the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) Study by Machine Learning. F1000Research. 2016. Data Source

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 31 Aug 2016

Author details Author details

Competing interests

PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests.

Grant information

Article Versions (3)

version 3

Revised

Published: 12 May 2017, 5:2124

https://doi.org/10.12688/f1000research.9417.3

version 2

Revised

Published: 27 Jan 2017, 5:2124

https://doi.org/10.12688/f1000research.9417.2

version 1

Published: 31 Aug 2016, 5:2124

https://doi.org/10.12688/f1000research.9417.1

© 2016 Rezaeian I et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Rezaeian I, Mucaki EJ, Baranova K et al. Predicting Outcomes of Hormone and Chemotherapy in the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) Study by Biochemically-inspired Machine Learning [version 1; peer review: 2 approved with reservations]. F1000Research 2016, 5:2124 (https://doi.org/10.12688/f1000research.9417.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 31 Aug 2016

Views

Reviewer Report 03 Oct 2016

Chun-Wei Tung, School of Pharmacy, Kaohsiung Medical University, Kaohsiung, Taiwan

Approved with Reservations

https://doi.org/10.5256/f1000research.10141.r16345

This study proposed prediction methods using SVM and RF classifiers with mRMR selected feature sets from cell line data and demonstrate its prediction ability for outcomes from METABRIC patient cohort. The classifiers with good prediction performance show the usefulness of combining domain knowledge with feature selection techniques. However, some details essential for reproducibility and interpretation are missing.
Required information is listed in the following.

What are the values of parameters for SVM and RF classifiers and the methods for parameter selection (by default or other selection methods)?
The development and evaluation of models for patient data are not clear. Whether the models were trained using partial data from METABRIC or only leave-one-out cross-validation was applied? If cross-validation is the case, then what is the model offered at the online server because there will be more than one models created, and whether the cross-validation is involved in the feature selection process that often leads to an overestimation of the performance. For the case of training on partial data, both training and test performance are essential information for evaluating the robustness of models.
Since some of the datasets are highly imbalanced, the numbers of positives and negatives, as well as sensitivity and specificity are more important than accuracy for interpreting the results as a high accuracy with a low AUC could be the result of all positive/negative predictions on an imbalanced dataset. Listing all the information along with the accuracy and AUC will help the interpretation of prediction performances.

Competing Interests: No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 27 Jan 2017

Peter Rogan, Department of Biochemistry, University of Western Ontario, London, Canada

27 Jan 2017

Author Response

Comment 1:What are the values of parameters for SVM and RF classifiers and the methods for parameter selection (by default or other selection methods)?

Response: The parameter values for ... Continue reading Comment 1:What are the values of parameters for SVM and RF classifiers and the methods for parameter selection (by default or other selection methods)?

Response: The parameter values for these classifiers have been added to the Tables 1-5.

In regards to parameter selection, the first paragraph of the methods now describes C and Sigma selection as a grid search to find the values with the lowest cross-validation misclassification rate. Similarly for RF, a grid search was used to optimize the maximum number of randomly selected genes for each tree (second paragraph of Methods section).

Comment 2: The development and evaluation of models for patient data are not clear. Whether the models were trained using partial data from METABRIC or only leave-one-out cross-validation was applied? If cross-validation is the case, then what is the model offered at the online server because there will be more than one models created, and whether the cross-validation is involved in the feature selection process that often leads to an overestimation of the performance. For the case of training on partial data, both training and test performance are essential information for evaluating the robustness of models.

Response: We obtained new results for both RF and mRMR+SVM models when we use discovery set as training set and validation set as test set, the performance of the model was poor. After more investigation we found that there happened to be a large variation between gene expression of 26 targeted genes between discovery and validation set (please see Supplementary Dataset 2). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance, since the training and test sets are vastly different.

However, we did carry out another experiment on discovery set solely and used 70% of data for training and remaining 30% for test the performance of the model. The results have been added to the manuscript (Tables 4 and 5).

Comment 3: Since some of the datasets are highly imbalanced, the numbers of positives and negatives, as well as sensitivity and specificity are more important than accuracy for interpreting the results as a high accuracy with a low AUC could be the result of all positive/negative predictions on an imbalanced dataset. Listing all the information along with the accuracy and AUC will help the interpretation of prediction performances.

Response: As previously mentioned, we have added more performance measures including MCC and AUC. They have been added Tables 1-5 of the manuscript.

Comment 1:What are the values of parameters for SVM and RF classifiers and the methods for parameter selection (by default or other selection methods)?

Response: The parameter values for these classifiers have been added to the Tables 1-5.

In regards to parameter selection, the first paragraph of the methods now describes C and Sigma selection as a grid search to find the values with the lowest cross-validation misclassification rate. Similarly for RF, a grid search was used to optimize the maximum number of randomly selected genes for each tree (second paragraph of Methods section).

Comment 2: The development and evaluation of models for patient data are not clear. Whether the models were trained using partial data from METABRIC or only leave-one-out cross-validation was applied? If cross-validation is the case, then what is the model offered at the online server because there will be more than one models created, and whether the cross-validation is involved in the feature selection process that often leads to an overestimation of the performance. For the case of training on partial data, both training and test performance are essential information for evaluating the robustness of models.

Response: We obtained new results for both RF and mRMR+SVM models when we use discovery set as training set and validation set as test set, the performance of the model was poor. After more investigation we found that there happened to be a large variation between gene expression of 26 targeted genes between discovery and validation set (please see Supplementary Dataset 2). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance, since the training and test sets are vastly different.

However, we did carry out another experiment on discovery set solely and used 70% of data for training and remaining 30% for test the performance of the model. The results have been added to the manuscript (Tables 4 and 5).

Comment 3: Since some of the datasets are highly imbalanced, the numbers of positives and negatives, as well as sensitivity and specificity are more important than accuracy for interpreting the results as a high accuracy with a low AUC could be the result of all positive/negative predictions on an imbalanced dataset. Listing all the information along with the accuracy and AUC will help the interpretation of prediction performances.

Response: As previously mentioned, we have added more performance measures including MCC and AUC. They have been added Tables 1-5 of the manuscript.
Competing Interests: PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 27 Jan 2017

Peter Rogan, Department of Biochemistry, University of Western Ontario, London, Canada

27 Jan 2017

Author Response

Comment 1:What are the values of parameters for SVM and RF classifiers and the methods for parameter selection (by default or other selection methods)?

Response: The parameter values for ... Continue reading Comment 1:What are the values of parameters for SVM and RF classifiers and the methods for parameter selection (by default or other selection methods)?

Response: The parameter values for these classifiers have been added to the Tables 1-5.

In regards to parameter selection, the first paragraph of the methods now describes C and Sigma selection as a grid search to find the values with the lowest cross-validation misclassification rate. Similarly for RF, a grid search was used to optimize the maximum number of randomly selected genes for each tree (second paragraph of Methods section).

Comment 2: The development and evaluation of models for patient data are not clear. Whether the models were trained using partial data from METABRIC or only leave-one-out cross-validation was applied? If cross-validation is the case, then what is the model offered at the online server because there will be more than one models created, and whether the cross-validation is involved in the feature selection process that often leads to an overestimation of the performance. For the case of training on partial data, both training and test performance are essential information for evaluating the robustness of models.

Response: We obtained new results for both RF and mRMR+SVM models when we use discovery set as training set and validation set as test set, the performance of the model was poor. After more investigation we found that there happened to be a large variation between gene expression of 26 targeted genes between discovery and validation set (please see Supplementary Dataset 2). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance, since the training and test sets are vastly different.

However, we did carry out another experiment on discovery set solely and used 70% of data for training and remaining 30% for test the performance of the model. The results have been added to the manuscript (Tables 4 and 5).

Comment 3: Since some of the datasets are highly imbalanced, the numbers of positives and negatives, as well as sensitivity and specificity are more important than accuracy for interpreting the results as a high accuracy with a low AUC could be the result of all positive/negative predictions on an imbalanced dataset. Listing all the information along with the accuracy and AUC will help the interpretation of prediction performances.

Response: As previously mentioned, we have added more performance measures including MCC and AUC. They have been added Tables 1-5 of the manuscript.

Comment 1:What are the values of parameters for SVM and RF classifiers and the methods for parameter selection (by default or other selection methods)?

Response: The parameter values for these classifiers have been added to the Tables 1-5.

In regards to parameter selection, the first paragraph of the methods now describes C and Sigma selection as a grid search to find the values with the lowest cross-validation misclassification rate. Similarly for RF, a grid search was used to optimize the maximum number of randomly selected genes for each tree (second paragraph of Methods section).

Comment 2: The development and evaluation of models for patient data are not clear. Whether the models were trained using partial data from METABRIC or only leave-one-out cross-validation was applied? If cross-validation is the case, then what is the model offered at the online server because there will be more than one models created, and whether the cross-validation is involved in the feature selection process that often leads to an overestimation of the performance. For the case of training on partial data, both training and test performance are essential information for evaluating the robustness of models.

Response: We obtained new results for both RF and mRMR+SVM models when we use discovery set as training set and validation set as test set, the performance of the model was poor. After more investigation we found that there happened to be a large variation between gene expression of 26 targeted genes between discovery and validation set (please see Supplementary Dataset 2). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance, since the training and test sets are vastly different.

However, we did carry out another experiment on discovery set solely and used 70% of data for training and remaining 30% for test the performance of the model. The results have been added to the manuscript (Tables 4 and 5).

Comment 3: Since some of the datasets are highly imbalanced, the numbers of positives and negatives, as well as sensitivity and specificity are more important than accuracy for interpreting the results as a high accuracy with a low AUC could be the result of all positive/negative predictions on an imbalanced dataset. Listing all the information along with the accuracy and AUC will help the interpretation of prediction performances.

Response: As previously mentioned, we have added more performance measures including MCC and AUC. They have been added Tables 1-5 of the manuscript.
Competing Interests: PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests. Close
Report a concern

Views

Reviewer Report 30 Sep 2016

Elana J. Fertig, Division of Oncology Biostatistics and Bioinformatics, School of Medicine, Johns Hopkins University, Baltimore, MD, USA

Approved with Reservations

https://doi.org/10.5256/f1000research.10141.r16733

This study develops SVM and RF algorithms built upon previously learned gene signatures of therapeutic response to breast cancer. The algorithms are applied and compared to predict patient survival under different treatment conditions in METABRIC data. The analyses and comparisons are robust and this study provides a useful assessment of biologically-driven classifiers. The three major areas that require improvement before the article is indexed are as follows, and described in further detail below.

The methods require further clarification to distinguish differences between this study and the previous study as well as the parameters of the machine learning algorithms.
Accuracy in the results must better distinguish results on independent test and training sets.
Classifiers must be put in the context of other existing genomics classifiers used in breast cancer and/or previously published in Mammaprint data.

Title and Abstract

Acceptable

Article content

Methods

Abbreviations SVM and RF must be spelled out as Support Vector Machine and Random Forrest on first use in Methods.
Writing in SVM learning subsection of Methods requires clarification to distinguish which of these methods were developed in the previous Molecular Oncology publication and which were developed as part of this publications.
Details about the SVM learning algorithm are included in the caption to Figure 1, but must also be included and completely described in text for the corresponding section of the methods.
No equations are provided to describe the role of the parameters C and sigma. It is also unclear whether this greedy search is implemented by the Matlab function fitcsvm or uses custom code developed by the authors.

Results

Need to specify whether reported accuracies are computed with leave-one-out cross validation or 9-fold cross validation (described in Methods).
Ideally, given the size of METABRIC data they would be calculated on independent training (first 1000 patient samples) and training (last 1000 patient samples) datasets.
AUC must be computed separately for discovery and validation sets (Table 2).
It is unclear whether the previous validation set described in the sentence “Predictions for the METABRIC patient cohort, which was independent of the previous validation set” refers to a validation set used in this publication or the previous publication.
Covariates such as ER/PR or PAM50 subtypes must be included in a table describing the sample cohorts. Accuracy must be computed separately for these co-variates or they must also be included as co-variates in the machine-learning model.
Ideally accuracy would be compared to existing breast cancer classifiers (e.g., using code from Marchionni et al., BMC Genomics, 2013) and/or survival curves reported in the literature.

Conclusions

Must be discussed in the context of existing genomics classifiers for breast cancer (e.g., OncotypeDx and/or Mammaprint).
Results must be put in context with other predictions on METABRIC data, e.g., outcomes from the DREAM contest.

Data

Acceptable

Competing Interests: No competing interests were disclosed.

CITE

Report a concern

Author Response 27 Jan 2017

Peter Rogan, Department of Biochemistry, University of Western Ontario, London, Canada

27 Jan 2017

Author Response

Comment 1: The methods require further clarification to distinguish differences between this study and the previous study as well as the parameters of the machine learning algorithms.

Response: The ... Continue reading Comment 1: The methods require further clarification to distinguish differences between this study and the previous study as well as the parameters of the machine learning algorithms.

Response: The first paragraph of the Methods describes Support Vector Machine learning, which has been greatly expanded upon. Differences in SVM methodology between the two studies are indicated there (i.e. a Gaussian kernel was used instead of a linear kernel). All other feature selection methods described in the manuscript (Random Forest, mRMR) were not used in Dorman et al., 2016.

The parameters for machine learning algorithms have been incorporated in the manuscript, and can be found in the footnote section of each data table.

Comment 2: Accuracy in the results must better distinguish results on independent test and training sets.

Response: The Validation dataset showed a distinct overall expression profile from the Discovery set, possibly due to batch effects, which are well known. We added another experiment to the manuscript by splitting the Discovery set into Training and Test sets. The model was trained using 70% of the data and then tested using the remaining 30% of data as test set. We repeated this procedure 100 times and took the median as the final performance result. The results are presented in Tables 4 and 5 of the manuscript.

Comment 3: Classifiers must be put in the context of other existing genomics classifiers used in breast cancer and/or previously published in Mammaprint data.

Response: We have added two sentences in the second paragraph of the “Results and Discussion” section which describes the comparison of our gene signature to those from MammaPrint and Oncotype Dx. Pair-wise comparison of these three signatures show that they are nearly independent of one another.

Methods

Comment 4: Abbreviations SVM and RF must be spelled out as Support Vector Machine and Random Forest on first use in Methods.

Response: We thank the reviewer for this suggestion. It has been addressed in the Methods section of the manuscript.

Comment 5: Writing in SVM learning subsection of Methods requires clarification to distinguish which of these methods were developed in the previous Molecular Oncology publication and which were developed as part of this publications.

Response: This is now clarified within the first paragraph of the Methods section in the manuscript. The SVM classifier was adopted from previous Molecular Oncology publication, while the feature selection method has been developed as part of this publication.

Comment 6: Details about the SVM learning algorithm are included in the caption to Figure 1, but must also be included and completely described in text for the corresponding section of the methods.

Response: Thanks for the reviewer’s suggestion. This description of the SVM learning algorithm has been moved from the Figure 1 legend and integrated into the first paragraph of the methods section.

Comment 7: No equations are provided to describe the role of the parameters C and sigma. It is also unclear whether this greedy search is implemented by the Matlab function fitcsvm or uses custom code developed by the authors.

Response: A brief description of the role of each parameter has been added to the first paragraph of the methods section of the manuscript. Readers are also now directed to a reference (Ben-Hur and Weston, 2010) if more detail is desired.

The greedy search, also called sequential backward feature selection, was implemented as a script by our lab in MATLAB. It is not a MATLAB function. This is clarified by changing a few words in the first paragraph of the methods section: “A backwards feature selection (greedy) algorithm was designed and implemented in MATLAB in which…”

Moreover, as described above, the SVM classifier was adopted from previous Molecular Oncology publication (Dorman et al. 2016), while the feature selection method has been developed as part of this publication.

Results
Comment 8: Need to specify whether reported accuracies are computed with leave-one-out cross validation or 9-fold cross validation (described in Methods).

Response: All SVM models described in the manuscript used leave-one-out cross validation except one, and this is clearly indicated in Table 1, and is now commented on in the methods. A 9-fold cross-validation was used to build a model using 735 patients who were treated with Chemotherapy and/or Hormone therapy, as leave-one-out cross validation of this many patients took an unreasonably long time to complete (it exceeded 3 weeks on a dedicated I7 Intel processor).

Comment 9: Ideally, given the size of METABRIC data they would be calculated on independent training (first 1000 patient samples) and test (last 1000 patient samples) datasets.

Response: We obtained new results for both RF and mRMR+SVM models using Discovery patient set for training and Validation set for testing, however the performance of the model was poor. After further investigation, we found that there were large differences between gene expression levels of the 26 model signature genes in the Discovery versus Validation sets (we used Wilcoxon rank sum test, Kruskal-Wallis test and t-test to evaluate the results – shown in the plotted distributions of gene expression in Supplemental Dataset 2) regardless of patient status (alive or dead). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance due to this source of heterogeneity.

To address this issue, we did carry out another experiment based on data from the Discovery patient dataset alone; using 70% of data for training and remaining 30% for testing, the performance of the model was significantly better. We speculate that the discrepancy between the expression distributions in the Discovery and Validation sets were the result of batch effects. The results have been added to the manuscript (Tables 4,5).

Comment 10: AUC must be computed separately for discovery and validation sets (Table 2).

Response: We have included additional performance measures to Tables 1-5, including Area Under Curve (AUC).

Comment 11: It is unclear whether the previous validation set described in the sentence “Predictions for the METABRIC patient cohort, which was independent of the previous validation set” refers to a validation set used in this publication or the previous publication.

Response: This sentence is referring to breast cancer patient data from Hatzis et al. (2013), which was used as a validation set in Dorman et al. (2016), not this publication. We have modified this sentence to clarify the issue.

Comment 12: Covariates such as ER/PR or PAM50 subtypes must be included in a table describing the sample cohorts. Accuracy must be computed separately for these co-variates or they must also be included as co-variates in the machine-learning model.

Response: Even with the subtype as covariant, it is not possible to perform the analysis the reviewer requested. Certain therapies are definitely more effective in particular subtypes (eg. etoposide, docetaxel, and cisplatin are preferentially active in basal or claudin-low cell lines, as observed clinically; Heiser et al., 2012). The public METABRIC dataset (or the corresponding publication) does not provide the specific therapies used to treat individual patients. Had they done so, it would have made sense to look at these covariates.
Reference: Heiser LM, Sadanandam A, Kuo WL, Benz SC, Goldstein TC, Ng S, Gibb WJ, Wang NJ, Ziyad S, Tong F, et al. (2012). Subtype and pathway specific responses to anticancer compounds in breast cancer. Proc Natl Acad Sci US A109:2724-2729.

Comment 13: Ideally accuracy would be compared to existing breast cancer classifiers (e.g., using code from Marchionni et al., BMC Genomics, 2013) and/or survival curves reported in the literature.

Response: The proposed method has been compared against the K-TSP (Marchionni et al., BMC Genomics, 2013) as per reviewer’s suggestion and the results are presented in Table 6 of the manuscript.

Conclusions
Comment 14: Must be discussed in the context of existing genomics classifiers for breast cancer (e.g., OncotypeDx and/or Mammaprint).

Response: We have added text to both the second paragraph of the “Results and Discussion” paragraph and to the conclusion of the paper.

Comment 15: Results must be put in context with other predictions on METABRIC data, e.g., outcomes from the DREAM contest.

Response: An important distinction to note in regards to our methodology is that the predictions are based on the genes known to be associated with the response to specific drugs used to treat breast cancer. In the DREAM contest, the method with the highest METABRIC score (as described in Cheng et al., 2013) was phenotype-based, finding signatures for molecular processes that are disregulated in METABRIC, rather than responses to the cancer therapies themselves. While this is an interesting prediction method, the results cannot compared to our approach. The gene signatures that we have derived contain components of many different pathways.
Reference: Cheng WY, Ou Yang TH, Anastassiou D. Biomolecular events in cancer revealed by attractor metagenes. PLoS Comput Biol. 2013;9(2):e1002920.
Comment 1: The methods require further clarification to distinguish differences between this study and the previous study as well as the parameters of the machine learning algorithms.

Response: The first paragraph of the Methods describes Support Vector Machine learning, which has been greatly expanded upon. Differences in SVM methodology between the two studies are indicated there (i.e. a Gaussian kernel was used instead of a linear kernel). All other feature selection methods described in the manuscript (Random Forest, mRMR) were not used in Dorman et al., 2016.

The parameters for machine learning algorithms have been incorporated in the manuscript, and can be found in the footnote section of each data table.

Comment 2: Accuracy in the results must better distinguish results on independent test and training sets.

Response: The Validation dataset showed a distinct overall expression profile from the Discovery set, possibly due to batch effects, which are well known. We added another experiment to the manuscript by splitting the Discovery set into Training and Test sets. The model was trained using 70% of the data and then tested using the remaining 30% of data as test set. We repeated this procedure 100 times and took the median as the final performance result. The results are presented in Tables 4 and 5 of the manuscript.

Comment 3: Classifiers must be put in the context of other existing genomics classifiers used in breast cancer and/or previously published in Mammaprint data.

Response: We have added two sentences in the second paragraph of the “Results and Discussion” section which describes the comparison of our gene signature to those from MammaPrint and Oncotype Dx. Pair-wise comparison of these three signatures show that they are nearly independent of one another.

Methods

Comment 4: Abbreviations SVM and RF must be spelled out as Support Vector Machine and Random Forest on first use in Methods.

Response: We thank the reviewer for this suggestion. It has been addressed in the Methods section of the manuscript.

Comment 5: Writing in SVM learning subsection of Methods requires clarification to distinguish which of these methods were developed in the previous Molecular Oncology publication and which were developed as part of this publications.

Response: This is now clarified within the first paragraph of the Methods section in the manuscript. The SVM classifier was adopted from previous Molecular Oncology publication, while the feature selection method has been developed as part of this publication.

Comment 6: Details about the SVM learning algorithm are included in the caption to Figure 1, but must also be included and completely described in text for the corresponding section of the methods.

Response: Thanks for the reviewer’s suggestion. This description of the SVM learning algorithm has been moved from the Figure 1 legend and integrated into the first paragraph of the methods section.

Comment 7: No equations are provided to describe the role of the parameters C and sigma. It is also unclear whether this greedy search is implemented by the Matlab function fitcsvm or uses custom code developed by the authors.

Response: A brief description of the role of each parameter has been added to the first paragraph of the methods section of the manuscript. Readers are also now directed to a reference (Ben-Hur and Weston, 2010) if more detail is desired.

The greedy search, also called sequential backward feature selection, was implemented as a script by our lab in MATLAB. It is not a MATLAB function. This is clarified by changing a few words in the first paragraph of the methods section: “A backwards feature selection (greedy) algorithm was designed and implemented in MATLAB in which…”

Moreover, as described above, the SVM classifier was adopted from previous Molecular Oncology publication (Dorman et al. 2016), while the feature selection method has been developed as part of this publication.

Results
Comment 8: Need to specify whether reported accuracies are computed with leave-one-out cross validation or 9-fold cross validation (described in Methods).

Response: All SVM models described in the manuscript used leave-one-out cross validation except one, and this is clearly indicated in Table 1, and is now commented on in the methods. A 9-fold cross-validation was used to build a model using 735 patients who were treated with Chemotherapy and/or Hormone therapy, as leave-one-out cross validation of this many patients took an unreasonably long time to complete (it exceeded 3 weeks on a dedicated I7 Intel processor).

Comment 9: Ideally, given the size of METABRIC data they would be calculated on independent training (first 1000 patient samples) and test (last 1000 patient samples) datasets.

Response: We obtained new results for both RF and mRMR+SVM models using Discovery patient set for training and Validation set for testing, however the performance of the model was poor. After further investigation, we found that there were large differences between gene expression levels of the 26 model signature genes in the Discovery versus Validation sets (we used Wilcoxon rank sum test, Kruskal-Wallis test and t-test to evaluate the results – shown in the plotted distributions of gene expression in Supplemental Dataset 2) regardless of patient status (alive or dead). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance due to this source of heterogeneity.

To address this issue, we did carry out another experiment based on data from the Discovery patient dataset alone; using 70% of data for training and remaining 30% for testing, the performance of the model was significantly better. We speculate that the discrepancy between the expression distributions in the Discovery and Validation sets were the result of batch effects. The results have been added to the manuscript (Tables 4,5).

Comment 10: AUC must be computed separately for discovery and validation sets (Table 2).

Response: We have included additional performance measures to Tables 1-5, including Area Under Curve (AUC).

Comment 11: It is unclear whether the previous validation set described in the sentence “Predictions for the METABRIC patient cohort, which was independent of the previous validation set” refers to a validation set used in this publication or the previous publication.

Response: This sentence is referring to breast cancer patient data from Hatzis et al. (2013), which was used as a validation set in Dorman et al. (2016), not this publication. We have modified this sentence to clarify the issue.

Comment 12: Covariates such as ER/PR or PAM50 subtypes must be included in a table describing the sample cohorts. Accuracy must be computed separately for these co-variates or they must also be included as co-variates in the machine-learning model.

Response: Even with the subtype as covariant, it is not possible to perform the analysis the reviewer requested. Certain therapies are definitely more effective in particular subtypes (eg. etoposide, docetaxel, and cisplatin are preferentially active in basal or claudin-low cell lines, as observed clinically; Heiser et al., 2012). The public METABRIC dataset (or the corresponding publication) does not provide the specific therapies used to treat individual patients. Had they done so, it would have made sense to look at these covariates.
Reference: Heiser LM, Sadanandam A, Kuo WL, Benz SC, Goldstein TC, Ng S, Gibb WJ, Wang NJ, Ziyad S, Tong F, et al. (2012). Subtype and pathway specific responses to anticancer compounds in breast cancer. Proc Natl Acad Sci US A109:2724-2729.

Comment 13: Ideally accuracy would be compared to existing breast cancer classifiers (e.g., using code from Marchionni et al., BMC Genomics, 2013) and/or survival curves reported in the literature.

Response: The proposed method has been compared against the K-TSP (Marchionni et al., BMC Genomics, 2013) as per reviewer’s suggestion and the results are presented in Table 6 of the manuscript.

Conclusions
Comment 14: Must be discussed in the context of existing genomics classifiers for breast cancer (e.g., OncotypeDx and/or Mammaprint).

Response: We have added text to both the second paragraph of the “Results and Discussion” paragraph and to the conclusion of the paper.

Comment 15: Results must be put in context with other predictions on METABRIC data, e.g., outcomes from the DREAM contest.

Response: An important distinction to note in regards to our methodology is that the predictions are based on the genes known to be associated with the response to specific drugs used to treat breast cancer. In the DREAM contest, the method with the highest METABRIC score (as described in Cheng et al., 2013) was phenotype-based, finding signatures for molecular processes that are disregulated in METABRIC, rather than responses to the cancer therapies themselves. While this is an interesting prediction method, the results cannot compared to our approach. The gene signatures that we have derived contain components of many different pathways.
Reference: Cheng WY, Ou Yang TH, Anastassiou D. Biomolecular events in cancer revealed by attractor metagenes. PLoS Comput Biol. 2013;9(2):e1002920.
Competing Interests: PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 27 Jan 2017

Peter Rogan, Department of Biochemistry, University of Western Ontario, London, Canada

27 Jan 2017

Author Response

Comment 1: The methods require further clarification to distinguish differences between this study and the previous study as well as the parameters of the machine learning algorithms.

Response: The ... Continue reading Comment 1: The methods require further clarification to distinguish differences between this study and the previous study as well as the parameters of the machine learning algorithms.

Response: The first paragraph of the Methods describes Support Vector Machine learning, which has been greatly expanded upon. Differences in SVM methodology between the two studies are indicated there (i.e. a Gaussian kernel was used instead of a linear kernel). All other feature selection methods described in the manuscript (Random Forest, mRMR) were not used in Dorman et al., 2016.

The parameters for machine learning algorithms have been incorporated in the manuscript, and can be found in the footnote section of each data table.

Comment 2: Accuracy in the results must better distinguish results on independent test and training sets.

Response: The Validation dataset showed a distinct overall expression profile from the Discovery set, possibly due to batch effects, which are well known. We added another experiment to the manuscript by splitting the Discovery set into Training and Test sets. The model was trained using 70% of the data and then tested using the remaining 30% of data as test set. We repeated this procedure 100 times and took the median as the final performance result. The results are presented in Tables 4 and 5 of the manuscript.

Comment 3: Classifiers must be put in the context of other existing genomics classifiers used in breast cancer and/or previously published in Mammaprint data.

Response: We have added two sentences in the second paragraph of the “Results and Discussion” section which describes the comparison of our gene signature to those from MammaPrint and Oncotype Dx. Pair-wise comparison of these three signatures show that they are nearly independent of one another.

Methods

Comment 4: Abbreviations SVM and RF must be spelled out as Support Vector Machine and Random Forest on first use in Methods.

Response: We thank the reviewer for this suggestion. It has been addressed in the Methods section of the manuscript.

Comment 5: Writing in SVM learning subsection of Methods requires clarification to distinguish which of these methods were developed in the previous Molecular Oncology publication and which were developed as part of this publications.

Response: This is now clarified within the first paragraph of the Methods section in the manuscript. The SVM classifier was adopted from previous Molecular Oncology publication, while the feature selection method has been developed as part of this publication.

Comment 6: Details about the SVM learning algorithm are included in the caption to Figure 1, but must also be included and completely described in text for the corresponding section of the methods.

Response: Thanks for the reviewer’s suggestion. This description of the SVM learning algorithm has been moved from the Figure 1 legend and integrated into the first paragraph of the methods section.

Comment 7: No equations are provided to describe the role of the parameters C and sigma. It is also unclear whether this greedy search is implemented by the Matlab function fitcsvm or uses custom code developed by the authors.

Response: A brief description of the role of each parameter has been added to the first paragraph of the methods section of the manuscript. Readers are also now directed to a reference (Ben-Hur and Weston, 2010) if more detail is desired.

The greedy search, also called sequential backward feature selection, was implemented as a script by our lab in MATLAB. It is not a MATLAB function. This is clarified by changing a few words in the first paragraph of the methods section: “A backwards feature selection (greedy) algorithm was designed and implemented in MATLAB in which…”

Moreover, as described above, the SVM classifier was adopted from previous Molecular Oncology publication (Dorman et al. 2016), while the feature selection method has been developed as part of this publication.

Results
Comment 8: Need to specify whether reported accuracies are computed with leave-one-out cross validation or 9-fold cross validation (described in Methods).

Response: All SVM models described in the manuscript used leave-one-out cross validation except one, and this is clearly indicated in Table 1, and is now commented on in the methods. A 9-fold cross-validation was used to build a model using 735 patients who were treated with Chemotherapy and/or Hormone therapy, as leave-one-out cross validation of this many patients took an unreasonably long time to complete (it exceeded 3 weeks on a dedicated I7 Intel processor).

Comment 9: Ideally, given the size of METABRIC data they would be calculated on independent training (first 1000 patient samples) and test (last 1000 patient samples) datasets.

Response: We obtained new results for both RF and mRMR+SVM models using Discovery patient set for training and Validation set for testing, however the performance of the model was poor. After further investigation, we found that there were large differences between gene expression levels of the 26 model signature genes in the Discovery versus Validation sets (we used Wilcoxon rank sum test, Kruskal-Wallis test and t-test to evaluate the results – shown in the plotted distributions of gene expression in Supplemental Dataset 2) regardless of patient status (alive or dead). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance due to this source of heterogeneity.

To address this issue, we did carry out another experiment based on data from the Discovery patient dataset alone; using 70% of data for training and remaining 30% for testing, the performance of the model was significantly better. We speculate that the discrepancy between the expression distributions in the Discovery and Validation sets were the result of batch effects. The results have been added to the manuscript (Tables 4,5).

Comment 10: AUC must be computed separately for discovery and validation sets (Table 2).

Response: We have included additional performance measures to Tables 1-5, including Area Under Curve (AUC).

Comment 11: It is unclear whether the previous validation set described in the sentence “Predictions for the METABRIC patient cohort, which was independent of the previous validation set” refers to a validation set used in this publication or the previous publication.

Response: This sentence is referring to breast cancer patient data from Hatzis et al. (2013), which was used as a validation set in Dorman et al. (2016), not this publication. We have modified this sentence to clarify the issue.

Comment 12: Covariates such as ER/PR or PAM50 subtypes must be included in a table describing the sample cohorts. Accuracy must be computed separately for these co-variates or they must also be included as co-variates in the machine-learning model.

Response: Even with the subtype as covariant, it is not possible to perform the analysis the reviewer requested. Certain therapies are definitely more effective in particular subtypes (eg. etoposide, docetaxel, and cisplatin are preferentially active in basal or claudin-low cell lines, as observed clinically; Heiser et al., 2012). The public METABRIC dataset (or the corresponding publication) does not provide the specific therapies used to treat individual patients. Had they done so, it would have made sense to look at these covariates.
Reference: Heiser LM, Sadanandam A, Kuo WL, Benz SC, Goldstein TC, Ng S, Gibb WJ, Wang NJ, Ziyad S, Tong F, et al. (2012). Subtype and pathway specific responses to anticancer compounds in breast cancer. Proc Natl Acad Sci US A109:2724-2729.

Comment 13: Ideally accuracy would be compared to existing breast cancer classifiers (e.g., using code from Marchionni et al., BMC Genomics, 2013) and/or survival curves reported in the literature.

Response: The proposed method has been compared against the K-TSP (Marchionni et al., BMC Genomics, 2013) as per reviewer’s suggestion and the results are presented in Table 6 of the manuscript.

Conclusions
Comment 14: Must be discussed in the context of existing genomics classifiers for breast cancer (e.g., OncotypeDx and/or Mammaprint).

Response: We have added text to both the second paragraph of the “Results and Discussion” paragraph and to the conclusion of the paper.

Comment 15: Results must be put in context with other predictions on METABRIC data, e.g., outcomes from the DREAM contest.

Response: An important distinction to note in regards to our methodology is that the predictions are based on the genes known to be associated with the response to specific drugs used to treat breast cancer. In the DREAM contest, the method with the highest METABRIC score (as described in Cheng et al., 2013) was phenotype-based, finding signatures for molecular processes that are disregulated in METABRIC, rather than responses to the cancer therapies themselves. While this is an interesting prediction method, the results cannot compared to our approach. The gene signatures that we have derived contain components of many different pathways.
Reference: Cheng WY, Ou Yang TH, Anastassiou D. Biomolecular events in cancer revealed by attractor metagenes. PLoS Comput Biol. 2013;9(2):e1002920.
Comment 1: The methods require further clarification to distinguish differences between this study and the previous study as well as the parameters of the machine learning algorithms.

Response: The first paragraph of the Methods describes Support Vector Machine learning, which has been greatly expanded upon. Differences in SVM methodology between the two studies are indicated there (i.e. a Gaussian kernel was used instead of a linear kernel). All other feature selection methods described in the manuscript (Random Forest, mRMR) were not used in Dorman et al., 2016.

The parameters for machine learning algorithms have been incorporated in the manuscript, and can be found in the footnote section of each data table.

Comment 2: Accuracy in the results must better distinguish results on independent test and training sets.

Response: The Validation dataset showed a distinct overall expression profile from the Discovery set, possibly due to batch effects, which are well known. We added another experiment to the manuscript by splitting the Discovery set into Training and Test sets. The model was trained using 70% of the data and then tested using the remaining 30% of data as test set. We repeated this procedure 100 times and took the median as the final performance result. The results are presented in Tables 4 and 5 of the manuscript.

Comment 3: Classifiers must be put in the context of other existing genomics classifiers used in breast cancer and/or previously published in Mammaprint data.

Response: We have added two sentences in the second paragraph of the “Results and Discussion” section which describes the comparison of our gene signature to those from MammaPrint and Oncotype Dx. Pair-wise comparison of these three signatures show that they are nearly independent of one another.

Methods

Comment 4: Abbreviations SVM and RF must be spelled out as Support Vector Machine and Random Forest on first use in Methods.

Response: We thank the reviewer for this suggestion. It has been addressed in the Methods section of the manuscript.

Comment 5: Writing in SVM learning subsection of Methods requires clarification to distinguish which of these methods were developed in the previous Molecular Oncology publication and which were developed as part of this publications.

Response: This is now clarified within the first paragraph of the Methods section in the manuscript. The SVM classifier was adopted from previous Molecular Oncology publication, while the feature selection method has been developed as part of this publication.

Comment 6: Details about the SVM learning algorithm are included in the caption to Figure 1, but must also be included and completely described in text for the corresponding section of the methods.

Response: Thanks for the reviewer’s suggestion. This description of the SVM learning algorithm has been moved from the Figure 1 legend and integrated into the first paragraph of the methods section.

Comment 7: No equations are provided to describe the role of the parameters C and sigma. It is also unclear whether this greedy search is implemented by the Matlab function fitcsvm or uses custom code developed by the authors.

Response: A brief description of the role of each parameter has been added to the first paragraph of the methods section of the manuscript. Readers are also now directed to a reference (Ben-Hur and Weston, 2010) if more detail is desired.

The greedy search, also called sequential backward feature selection, was implemented as a script by our lab in MATLAB. It is not a MATLAB function. This is clarified by changing a few words in the first paragraph of the methods section: “A backwards feature selection (greedy) algorithm was designed and implemented in MATLAB in which…”

Moreover, as described above, the SVM classifier was adopted from previous Molecular Oncology publication (Dorman et al. 2016), while the feature selection method has been developed as part of this publication.

Results
Comment 8: Need to specify whether reported accuracies are computed with leave-one-out cross validation or 9-fold cross validation (described in Methods).

Response: All SVM models described in the manuscript used leave-one-out cross validation except one, and this is clearly indicated in Table 1, and is now commented on in the methods. A 9-fold cross-validation was used to build a model using 735 patients who were treated with Chemotherapy and/or Hormone therapy, as leave-one-out cross validation of this many patients took an unreasonably long time to complete (it exceeded 3 weeks on a dedicated I7 Intel processor).

Comment 9: Ideally, given the size of METABRIC data they would be calculated on independent training (first 1000 patient samples) and test (last 1000 patient samples) datasets.

Response: We obtained new results for both RF and mRMR+SVM models using Discovery patient set for training and Validation set for testing, however the performance of the model was poor. After further investigation, we found that there were large differences between gene expression levels of the 26 model signature genes in the Discovery versus Validation sets (we used Wilcoxon rank sum test, Kruskal-Wallis test and t-test to evaluate the results – shown in the plotted distributions of gene expression in Supplemental Dataset 2) regardless of patient status (alive or dead). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance due to this source of heterogeneity.

To address this issue, we did carry out another experiment based on data from the Discovery patient dataset alone; using 70% of data for training and remaining 30% for testing, the performance of the model was significantly better. We speculate that the discrepancy between the expression distributions in the Discovery and Validation sets were the result of batch effects. The results have been added to the manuscript (Tables 4,5).

Comment 10: AUC must be computed separately for discovery and validation sets (Table 2).

Response: We have included additional performance measures to Tables 1-5, including Area Under Curve (AUC).

Comment 11: It is unclear whether the previous validation set described in the sentence “Predictions for the METABRIC patient cohort, which was independent of the previous validation set” refers to a validation set used in this publication or the previous publication.

Response: This sentence is referring to breast cancer patient data from Hatzis et al. (2013), which was used as a validation set in Dorman et al. (2016), not this publication. We have modified this sentence to clarify the issue.

Comment 12: Covariates such as ER/PR or PAM50 subtypes must be included in a table describing the sample cohorts. Accuracy must be computed separately for these co-variates or they must also be included as co-variates in the machine-learning model.

Response: Even with the subtype as covariant, it is not possible to perform the analysis the reviewer requested. Certain therapies are definitely more effective in particular subtypes (eg. etoposide, docetaxel, and cisplatin are preferentially active in basal or claudin-low cell lines, as observed clinically; Heiser et al., 2012). The public METABRIC dataset (or the corresponding publication) does not provide the specific therapies used to treat individual patients. Had they done so, it would have made sense to look at these covariates.
Reference: Heiser LM, Sadanandam A, Kuo WL, Benz SC, Goldstein TC, Ng S, Gibb WJ, Wang NJ, Ziyad S, Tong F, et al. (2012). Subtype and pathway specific responses to anticancer compounds in breast cancer. Proc Natl Acad Sci US A109:2724-2729.

Comment 13: Ideally accuracy would be compared to existing breast cancer classifiers (e.g., using code from Marchionni et al., BMC Genomics, 2013) and/or survival curves reported in the literature.

Response: The proposed method has been compared against the K-TSP (Marchionni et al., BMC Genomics, 2013) as per reviewer’s suggestion and the results are presented in Table 6 of the manuscript.

Conclusions
Comment 14: Must be discussed in the context of existing genomics classifiers for breast cancer (e.g., OncotypeDx and/or Mammaprint).

Response: We have added text to both the second paragraph of the “Results and Discussion” paragraph and to the conclusion of the paper.

Comment 15: Results must be put in context with other predictions on METABRIC data, e.g., outcomes from the DREAM contest.

Response: An important distinction to note in regards to our methodology is that the predictions are based on the genes known to be associated with the response to specific drugs used to treat breast cancer. In the DREAM contest, the method with the highest METABRIC score (as described in Cheng et al., 2013) was phenotype-based, finding signatures for molecular processes that are disregulated in METABRIC, rather than responses to the cancer therapies themselves. While this is an interesting prediction method, the results cannot compared to our approach. The gene signatures that we have derived contain components of many different pathways.
Reference: Cheng WY, Ou Yang TH, Anastassiou D. Biomolecular events in cancer revealed by attractor metagenes. PLoS Comput Biol. 2013;9(2):e1002920.
Competing Interests: PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests. Close
Report a concern

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 31 Aug 2016

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 3 (revision) 12 May 17	read
Version 2 (revision) 27 Jan 17	read	read
Version 1 31 Aug 16	read	read

Elana J. Fertig, Johns Hopkins University, Baltimore, USA
Chun-Wei Tung, Kaohsiung Medical University, Kaohsiung, Taiwan

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

18 Views

31 May 2017 | for Version 3

Elana J. Fertig, Division of Oncology Biostatistics and Bioinformatics, School of Medicine, Johns Hopkins University, Baltimore, MD, USA

18 Views Cite this report Responses(0)

Approved

The authors have addressed all previous comments.

Competing Interests

No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

25 Views

21 Feb 2017 | for Version 2

Chun-Wei Tung, School of Pharmacy, Kaohsiung Medical University, Kaohsiung, Taiwan

25 Views Cite this report Responses(1)

Approved

The authors have addressed all the concerns raised from the previous review. A minor comment for the batch effects is given in the follows. As batch effects are expected for heterogeneous datasets, the direct application of prediction model built on the discovery dataset to the validation dataset would be incorrect and usually result in poor performance. To show the usefulness of the gene signatures while minimizing the batch effects, the authors might consider to run cross-validation on the validation dataset alone using the gene signatures obtained from discovery dataset.

Competing Interests

No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

12 May 2017

Peter Rogan, CytoGnomix Inc, London, Canada

Thank you for your suggestion. As recommended, we have repeated the 70/30% cross-validation analysis performed in the manuscript (Tables 4 and 5) with the same genes obtained from the Discovery dataset (Tables 4 and 5), but using the Validation dataset alone for training and testing. We found that this analysis had a similar performance level as the analysis reported in the main manuscript (Tables 4 and 5). There are exceptions. The mRMR+SVM gene signature developed using “CT-only” patients at a 5-year threshold was much less accurate using the Validation data. However, the “CT-only” subset of the Validation dataset is small (N=31), and thus variability is not unexpected. Overall, this analysis suggests that the cross-validation issue was indeed mostly due to batch effects.

The following sentence was written in the main text which describes this result:

“Starting with the same set of Discovery genes, we also trained a separate model using the Validation data, and tested this data by 70/30% cross-validation (accuracy for RF: 56-67% [CT], 67-83% [HT], 56-81% [CT-HT]; accuracy for mRMR: 33-56% [CT], 70-84% [HT], 64-82% [CT-HT]).”

View more View less

Competing Interests

PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests.

Back to all reports

Reviewer Report

44 Views

03 Feb 2017 | for Version 2

Elana J. Fertig, Division of Oncology Biostatistics and Bioinformatics, School of Medicine, Johns Hopkins University, Baltimore, MD, USA

44 Views Cite this report Responses(1)

Approved With Reservations

The authors were very responsive to the previous round of reviews, including more robust cross-validation and cross-study validation and comparison with other classifiers. Particular concerns remain that the author’s conclusions that it is inappropriate to perform cross-study validation due to batch effects are incorrect, particularly since this challenging task is essential to assess overfitting and for clinical translation of classifiers. In addition, the conclusion was insufficiently revised to place their classifier in the context of the broader literature in this field.

Methods

Abbreviations SVM and RF must be spelled out as Support Vector Machine and Random Forest on first use. This was not addressed in the revised methods section.

Results

The authors did perform a robust cross-study validation, as requested in the previous review. We agree this is challenging, due in part to batch effects as reported in this manuscript. However, such cross-study validation is essential to assess the accuracy of classifiers. It is also essential to have translation of genomic signatures into the clinic, where even different assays may be used. To address these concerns the authors must do the following: (a) Remove the sentence “This heterogeneity indicates that it is inappropriate to test our gene expression signatures derived by one of these datasets using the other dataset.” (b) Discuss the importance of cross-study validation, challenges in this application, and potential of overfitting of suggested by these results.
The author’s response that specific therapies were not provided in METABRIC is incorrect. According to Curtis et al., (2012) “Nearly all oestrogen receptor (ER)-positive and/or lymph node (LN)-negative patients did not receive chemotherapy, whereas ER-negative and LN-positive patients did. Additionally, none of the HER2⁺ patients received trastuzumab. As such, the treatments were homogeneous with respect to clinically relevant groupings.” Therefore, the previous criticism #12 remains. Covariates such as ER/HER2/LN or PAM50 subtypes must be included in a table describing the sample cohorts remains. In addition, accuracy must be computed separately for these co-variates or included in the machine learning model.

Conclusion

The discussion is insufficient. It still lacks sufficient context of existing genomics classifiers in the literature. The discrepancy between their algorithm and clinical assays is confusing in revised sentence “Unlike Mammaprint and Oncotype Dx tests, this model focuses on predicting survival prediction based on gene expression in the tumor, presumably before or during drug therapy.” As written, it appears to disregard the long history of predicting clinical outcome from gene expression involved in developing these classifiers from gene expression data (e.g., van't Veer et al., 2002) into clinical assays based upon expression of smaller numbers of genes.
Based on the previous review, the authors include context with other predictions of the METABRIC data in the response to the reviewers. This must also be included in the Conclusion to assess the relevance of their findings in the literature.

References

1. Curtis C, Shah SP, Chin SF, Turashvili G, et al.: The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups.Nature. 2012; 486 (7403): 346-52 PubMed Abstract | Publisher Full Text
2. van 't Veer LJ, Dai H, van de Vijver MJ, He YD, et al.: Gene expression profiling predicts clinical outcome of breast cancer.Nature. 2002; 415 (6871): 530-6 PubMed Abstract | Publisher Full Text

Competing Interests

No competing interests were disclosed.

Respond to this report

Responses (1)

Author Response

12 May 2017

Peter Rogan, CytoGnomix Inc, London, Canada

Methods
Comment 1. Abbreviations SVM and RF must be spelled out as Support Vector Machine and
Random Forest on first use. This was not addressed in the revised methods section.

Response: These abbreviations are now spelled out upon their first use in the main text (Methods section).

Results
Comment 2. The authors did perform a robust cross-study validation, as requested in the previous review. We agree this is challenging, due in part to batch effects as reported in this manuscript. However, such cross-study validation is essential to assess the accuracy of classifiers. It is also essential to have translation of genomic signatures into the clinic, where even different assays may be used. To address these concerns the authors must do the following: (a) Remove the sentence “This heterogeneity indicates that it is inappropriate to test our gene expression signatures derived by one of these datasets using the other dataset.” (b) Discuss the importance of cross-study validation, challenges in this application, and potential of overfitting of suggested by these results.

Response: In regards to this point:
(a) This sentence has been removed, as requested.
(b) To address concerns regarding potential overfitting of our models, we cross-validate the acquired models to a non-METABRIC data set (from an independent study). In the Sage Bionetworks / DREAM Breast Cancer Prognosis Challenge, cross-study validation was performed using the “OsloVal” data set, which consists of gene expression and copy number data from 184 breast cancer patients (Margolin et al., 2013). However, this dataset is not publically available and requires Ethics Board / IRB Review which we did not believe to be worth the effort. Instead, we performed cross-study validation on the gene expression of 310 breast cancer patients made publically available by Hatzis et al. (2011).

Analysis of this dataset was successful for the mRMR + SVM models developed using chemotherapy-treated patient (“CT” models), where the threshold for resistance was set to 3-years and 4-years. The “CT 3-year” model performed well predicting responsive patients (74.2% accuracy), while the “CT 4-year” model performed better predicting non-responsive patients (75.1% accuracy). The “CT 4-year” model outperformed the “CT 5-year” model for both sensitive and resistant patient data sets.

Random Forest and mRMR+SVM models which used hormone-treated patients (“HT” and “CT+HT”) were much less accurate compared to the “CT-only” models, and predict patients a large percentage of patients from the Hatzis data as sensitive.

In the main manuscript, we have replaced the removed sentence from (a) and have written the following:
“Cross-study validation allows for the comparison of classification accuracy between the generated gene signatures. The observed heterogeneity in gene expression highlights one of the many challenges of cross-validation of gene signatures between these data from the same study exhibit drastic differences (for example, BCL2L1; Supplementary file 2). Furthermore, these gene expression differences also affect the performance of these methods when these datasets were combined (compare Table 2 and Table 4 for RF; Table 3 and Table 5 for mRMR). We considered the possibility that the Discovery model might be subject to overfitting. We therefore performed cross-study validation of the Discovery set-signature with an independently-derived dataset (319 invasive breast cancer patients treated with paclitaxel and anthracycline chemotherapy⁵). The mRMR+SVM CT-models performed well (4-year threshold model had an overall accuracy of 68.7%; 3-year threshold model exhibited lower overall accuracy [52%], but was significantly better at predicting patients in remission [74.2%]).”

References
Margolin AA, et al. Systematic analysis of challenge-driven improvements in molecular prognostic models for breast cancer. Sci Transl Med. 2013 Apr 17;5(181):181re1. doi: 10.1126/scitranslmed.3006112.
Hatzis, C., et al. 2011. A genomic predictor of response and survival following taxane-anthracycline chemotherapy for invasive breast cancer. JAMA. 305, 1873–1881.

Comment 3. The author’s response that specific therapies were not provided in METABRIC is incorrect. According to Curtis et al., (2012) “Nearly all oestrogen receptor (ER)-positive and/or lymph node (LN)-negative patients did not receive chemotherapy, whereas ER-negative and LN-positive patients did. Additionally, none of the HER2⁺ patients received trastuzumab. As such, the treatments were homogeneous with respect to clinically relevant groupings.” Therefore, the previous criticism #12 remains. Covariates such as ER/HER2/LN or PAM50 subtypes must be included in a table describing the sample cohorts remains. In addition, accuracy must be computed separately for these co-variates or included in the machine learning model.

Response: Thank you for the clarification regarding patient treatment. As a response, we have added an additional supplementary table which breaks down the accuracy of our models by subtype (ER, HER2, PR, LN and PAM50; Dataset 2). In the main text, we note that accuracy of most models are consistent between subtypes (+/- 10% deviation in accuracy). Subtypes with less than twenty individuals were ignored due to its small sample size. The following deviations in accuracy were noted:

Random Forest and mRMR models are shown to be consistently more accurate in predicting ER+, HER2- when treated with hormone therapy (both “HT” and “CT and/or HT” categories), when compared to ER- and HER2+ patients. The PAM50 basal subtype is consistently low in accuracy when testing patients treated with hormone therapy. This is most likely partially influenced by the RF and mRMR models for ‘HT’ to more often predict patients as sensitive, combined with the fact that ER+ and HER2- patients were more likely to response to therapy. It is important to note that the accuracy of predictions by RF and mRMR with patients treated only with chemotherapy was fairly consistent across all available subtypes (+/- 10% accuracy).
SVM paclitaxel models performed significantly better with HER2+ patients (26 correct, 3 misclassified; 90% accurate) in HER2- patients (40 correct, 15 misclassified; 73% accurate) when tested on patients treated with both hormone and chemotherapy. In Dorman et al (2016), it was stated that MAPT expression (which is present in the paclitaxel model) segregated with PAM50 luminal and basal subtypes. For this model, the accuracies of these subtypes are nearly identical to the accuracy of the entire subset.

Text describing these results can be found in the third paragraph of the results.

Conclusion
Comment 4. The discussion is insufficient. It still lacks sufficient context of existing genomics classifiers in the literature. The discrepancy between their algorithm and clinical assays is confusing in revised sentence:
“Unlike Mammaprint and Oncotype Dx tests, this model focuses on predicting survival prediction based on gene expression in the tumor, presumably before or during drug therapy.”
As written, it appears to disregard the long history of predicting clinical outcome from gene expression involved in developing these classifiers from gene expression data (e.g., van't Veer et al., 2002) into clinical assays based upon expression of smaller numbers of genes.

Response: We have removed the indicated sentence, which we agree was insufficient to the comment from the previous iteration of this article: “Must be discussed in the context of existing genomics classifiers for breast cancer (e.g., OncotypeDx and/or Mammaprint)”.

We in no way meant to ignore the long history of predicting clinical outcome from gene expression (as well as other genomic factors). A discussion on this topic was not included in earlier submissions as it initially had an imposed word length limit (upon first submission). We did, however, reference other articles which do discuss this topic. In Dorman et al. (2016), which described some of the methodology for initial gene selection that this study was based on, these contributions are well-referenced, including the history of the prediction of clinical outcome from genomic status:
“Previous studies have derived associations between the genomic status of one or more genes and tumor response to certain therapies (Duan et al., 2003; Glinsky et al., 2005; Hatzis et al., 2011; Ma et al., 2004; Rajput et al., 2013; van't Veer et al., 2002).

Correlations between single gene expression and tumor resistance (Duan et al., 2003, 1999) do not take into account multiple mechanisms of resistance or assess interactions between multiple genes. ABC transporter overexpression has long been shown to confer resistance, but enzymatic or functional inhibition has not substantially improve patient response to chemotherapy (Samuels et al., 1997).

Multi-gene analytical approaches have previously been successful in deriving prognostic gene signatures for metastatic risk stratification (Oncotype DX™, MammaPrint^®), subtypes (PAM50), and efforts have been made to predict chemotherapy resistance (Hess et al., 2006; Hatzis et al., 2011). “

In response to Dr. Fertig’s comments, we have added a short discussion with citations of previously published approaches (including MammaPrint and Oncotype DX):
“Genomic information has been shown to correlate with tumor therapy response in previous studies^5,12-16. From these studies, analytical methods have been used to develop gene signatures for chemotherapy resistance prediction⁵, subtypes (PAM50), and metastatic risk stratification (Oncotype DX™, MammaPrint^®).”

Comment 5. Based on the previous review, the authors include context with other predictions of the METABRIC data in the response to the reviewers. This must also be included in the Conclusion to assess the relevance of their findings in the literature.

Response: We have added the indicated text from the previous ‘response to the reviewers’ (modified) to the Conclusions:

“We also examined the method exhibiting the best performance in the Sage Bionetworks / DREAM Breast Cancer Prognosis Challenge¹⁷, which was also phenotype-based, however it produces outcome signatures based on molecular processes, rather than the cancer drugs themselves. While interesting and informative, the results cannot be directly compared.”

Please note that the majority of entries in the DREAM project were not fully curated and only exist as source code. Analyzing these files to determine what methodology was attempted by these groups is beyond the scope of our study. A description of the second place of the METABRIC phase of the DREAM challenge is provided in the link below. This link describes how the METABRIC data is trained using a bipartite graphing as input for linear models, boost models, and RankSVM. While they state that RankSVM was the least successful between the three methods, it does not appear that this particular study has been published to the literature. As a result, we cannot fully review their results, and thus cannot be compared to our methodology in the main manuscript.

https://sagesynapse.wordpress.com/2012/11/01/breast-cancer-challenge-team-pitttransmed-places-second-for-metabric-phase-of-the-challenge/

View more View less

Competing Interests

PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests.

Back to all reports

Reviewer Report

37 Views

03 Oct 2016 | for Version 1

Chun-Wei Tung, School of Pharmacy, Kaohsiung Medical University, Kaohsiung, Taiwan

37 Views Cite this report Responses(1)

Approved With Reservations

What are the values of parameters for SVM and RF classifiers and the methods for parameter selection (by default or other selection methods)?
The development and evaluation of models for patient data are not clear. Whether the models were trained using partial data from METABRIC or only leave-one-out cross-validation was applied? If cross-validation is the case, then what is the model offered at the online server because there will be more than one models created, and whether the cross-validation is involved in the feature selection process that often leads to an overestimation of the performance. For the case of training on partial data, both training and test performance are essential information for evaluating the robustness of models.
Since some of the datasets are highly imbalanced, the numbers of positives and negatives, as well as sensitivity and specificity are more important than accuracy for interpreting the results as a high accuracy with a low AUC could be the result of all positive/negative predictions on an imbalanced dataset. Listing all the information along with the accuracy and AUC will help the interpretation of prediction performances.

Competing Interests

No competing interests were disclosed.

Respond to this report

Responses (1)

Author Response

27 Jan 2017

Peter Rogan, Department of Biochemistry, University of Western Ontario, London, Canada

Comment 1:What are the values of parameters for SVM and RF classifiers and the methods for parameter selection (by default or other selection methods)?

Response: The parameter values for these classifiers have been added to the Tables 1-5.

In regards to parameter selection, the first paragraph of the methods now describes C and Sigma selection as a grid search to find the values with the lowest cross-validation misclassification rate. Similarly for RF, a grid search was used to optimize the maximum number of randomly selected genes for each tree (second paragraph of Methods section).

Comment 2: The development and evaluation of models for patient data are not clear. Whether the models were trained using partial data from METABRIC or only leave-one-out cross-validation was applied? If cross-validation is the case, then what is the model offered at the online server because there will be more than one models created, and whether the cross-validation is involved in the feature selection process that often leads to an overestimation of the performance. For the case of training on partial data, both training and test performance are essential information for evaluating the robustness of models.

Response: We obtained new results for both RF and mRMR+SVM models when we use discovery set as training set and validation set as test set, the performance of the model was poor. After more investigation we found that there happened to be a large variation between gene expression of 26 targeted genes between discovery and validation set (please see Supplementary Dataset 2). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance, since the training and test sets are vastly different.

However, we did carry out another experiment on discovery set solely and used 70% of data for training and remaining 30% for test the performance of the model. The results have been added to the manuscript (Tables 4 and 5).

Comment 3: Since some of the datasets are highly imbalanced, the numbers of positives and negatives, as well as sensitivity and specificity are more important than accuracy for interpreting the results as a high accuracy with a low AUC could be the result of all positive/negative predictions on an imbalanced dataset. Listing all the information along with the accuracy and AUC will help the interpretation of prediction performances.

Response: As previously mentioned, we have added more performance measures including MCC and AUC. They have been added Tables 1-5 of the manuscript.

View more View less

Competing Interests

PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests.

Back to all reports

Reviewer Report

46 Views

30 Sep 2016 | for Version 1

Elana J. Fertig, Division of Oncology Biostatistics and Bioinformatics, School of Medicine, Johns Hopkins University, Baltimore, MD, USA

46 Views Cite this report Responses(1)

Approved With Reservations

The methods require further clarification to distinguish differences between this study and the previous study as well as the parameters of the machine learning algorithms.
Accuracy in the results must better distinguish results on independent test and training sets.
Classifiers must be put in the context of other existing genomics classifiers used in breast cancer and/or previously published in Mammaprint data.

Title and Abstract

Acceptable

Article content

Methods

Abbreviations SVM and RF must be spelled out as Support Vector Machine and Random Forrest on first use in Methods.
Writing in SVM learning subsection of Methods requires clarification to distinguish which of these methods were developed in the previous Molecular Oncology publication and which were developed as part of this publications.
Details about the SVM learning algorithm are included in the caption to Figure 1, but must also be included and completely described in text for the corresponding section of the methods.
No equations are provided to describe the role of the parameters C and sigma. It is also unclear whether this greedy search is implemented by the Matlab function fitcsvm or uses custom code developed by the authors.

Results

Need to specify whether reported accuracies are computed with leave-one-out cross validation or 9-fold cross validation (described in Methods).
Ideally, given the size of METABRIC data they would be calculated on independent training (first 1000 patient samples) and training (last 1000 patient samples) datasets.
AUC must be computed separately for discovery and validation sets (Table 2).
It is unclear whether the previous validation set described in the sentence “Predictions for the METABRIC patient cohort, which was independent of the previous validation set” refers to a validation set used in this publication or the previous publication.
Covariates such as ER/PR or PAM50 subtypes must be included in a table describing the sample cohorts. Accuracy must be computed separately for these co-variates or they must also be included as co-variates in the machine-learning model.
Ideally accuracy would be compared to existing breast cancer classifiers (e.g., using code from Marchionni et al., BMC Genomics, 2013) and/or survival curves reported in the literature.

Conclusions

Must be discussed in the context of existing genomics classifiers for breast cancer (e.g., OncotypeDx and/or Mammaprint).
Results must be put in context with other predictions on METABRIC data, e.g., outcomes from the DREAM contest.

Data

Acceptable

Competing Interests

No competing interests were disclosed.

Respond to this report

Responses (1)

Author Response

27 Jan 2017

Peter Rogan, Department of Biochemistry, University of Western Ontario, London, Canada

Comment 1: The methods require further clarification to distinguish differences between this study and the previous study as well as the parameters of the machine learning algorithms.

Response: The first paragraph of the Methods describes Support Vector Machine learning, which has been greatly expanded upon. Differences in SVM methodology between the two studies are indicated there (i.e. a Gaussian kernel was used instead of a linear kernel). All other feature selection methods described in the manuscript (Random Forest, mRMR) were not used in Dorman et al., 2016.

The parameters for machine learning algorithms have been incorporated in the manuscript, and can be found in the footnote section of each data table.

Comment 2: Accuracy in the results must better distinguish results on independent test and training sets.

Response: The Validation dataset showed a distinct overall expression profile from the Discovery set, possibly due to batch effects, which are well known. We added another experiment to the manuscript by splitting the Discovery set into Training and Test sets. The model was trained using 70% of the data and then tested using the remaining 30% of data as test set. We repeated this procedure 100 times and took the median as the final performance result. The results are presented in Tables 4 and 5 of the manuscript.

Comment 3: Classifiers must be put in the context of other existing genomics classifiers used in breast cancer and/or previously published in Mammaprint data.

Response: We have added two sentences in the second paragraph of the “Results and Discussion” section which describes the comparison of our gene signature to those from MammaPrint and Oncotype Dx. Pair-wise comparison of these three signatures show that they are nearly independent of one another.

Methods

Comment 4: Abbreviations SVM and RF must be spelled out as Support Vector Machine and Random Forest on first use in Methods.

Response: We thank the reviewer for this suggestion. It has been addressed in the Methods section of the manuscript.

Comment 5: Writing in SVM learning subsection of Methods requires clarification to distinguish which of these methods were developed in the previous Molecular Oncology publication and which were developed as part of this publications.

Response: This is now clarified within the first paragraph of the Methods section in the manuscript. The SVM classifier was adopted from previous Molecular Oncology publication, while the feature selection method has been developed as part of this publication.

Comment 6: Details about the SVM learning algorithm are included in the caption to Figure 1, but must also be included and completely described in text for the corresponding section of the methods.

Response: Thanks for the reviewer’s suggestion. This description of the SVM learning algorithm has been moved from the Figure 1 legend and integrated into the first paragraph of the methods section.

Comment 7: No equations are provided to describe the role of the parameters C and sigma. It is also unclear whether this greedy search is implemented by the Matlab function fitcsvm or uses custom code developed by the authors.

Response: A brief description of the role of each parameter has been added to the first paragraph of the methods section of the manuscript. Readers are also now directed to a reference (Ben-Hur and Weston, 2010) if more detail is desired.

The greedy search, also called sequential backward feature selection, was implemented as a script by our lab in MATLAB. It is not a MATLAB function. This is clarified by changing a few words in the first paragraph of the methods section: “A backwards feature selection (greedy) algorithm was designed and implemented in MATLAB in which…”

Moreover, as described above, the SVM classifier was adopted from previous Molecular Oncology publication (Dorman et al. 2016), while the feature selection method has been developed as part of this publication.

Results
Comment 8: Need to specify whether reported accuracies are computed with leave-one-out cross validation or 9-fold cross validation (described in Methods).

Response: All SVM models described in the manuscript used leave-one-out cross validation except one, and this is clearly indicated in Table 1, and is now commented on in the methods. A 9-fold cross-validation was used to build a model using 735 patients who were treated with Chemotherapy and/or Hormone therapy, as leave-one-out cross validation of this many patients took an unreasonably long time to complete (it exceeded 3 weeks on a dedicated I7 Intel processor).

Comment 9: Ideally, given the size of METABRIC data they would be calculated on independent training (first 1000 patient samples) and test (last 1000 patient samples) datasets.

Response: We obtained new results for both RF and mRMR+SVM models using Discovery patient set for training and Validation set for testing, however the performance of the model was poor. After further investigation, we found that there were large differences between gene expression levels of the 26 model signature genes in the Discovery versus Validation sets (we used Wilcoxon rank sum test, Kruskal-Wallis test and t-test to evaluate the results – shown in the plotted distributions of gene expression in Supplemental Dataset 2) regardless of patient status (alive or dead). Hence, building any classifier using discovery and validation set as training and test set in their current forms will result of poor performance due to this source of heterogeneity.

To address this issue, we did carry out another experiment based on data from the Discovery patient dataset alone; using 70% of data for training and remaining 30% for testing, the performance of the model was significantly better. We speculate that the discrepancy between the expression distributions in the Discovery and Validation sets were the result of batch effects. The results have been added to the manuscript (Tables 4,5).

Comment 10: AUC must be computed separately for discovery and validation sets (Table 2).

Response: We have included additional performance measures to Tables 1-5, including Area Under Curve (AUC).

Comment 11: It is unclear whether the previous validation set described in the sentence “Predictions for the METABRIC patient cohort, which was independent of the previous validation set” refers to a validation set used in this publication or the previous publication.

Response: This sentence is referring to breast cancer patient data from Hatzis et al. (2013), which was used as a validation set in Dorman et al. (2016), not this publication. We have modified this sentence to clarify the issue.

Comment 12: Covariates such as ER/PR or PAM50 subtypes must be included in a table describing the sample cohorts. Accuracy must be computed separately for these co-variates or they must also be included as co-variates in the machine-learning model.

Response: Even with the subtype as covariant, it is not possible to perform the analysis the reviewer requested. Certain therapies are definitely more effective in particular subtypes (eg. etoposide, docetaxel, and cisplatin are preferentially active in basal or claudin-low cell lines, as observed clinically; Heiser et al., 2012). The public METABRIC dataset (or the corresponding publication) does not provide the specific therapies used to treat individual patients. Had they done so, it would have made sense to look at these covariates.
Reference: Heiser LM, Sadanandam A, Kuo WL, Benz SC, Goldstein TC, Ng S, Gibb WJ, Wang NJ, Ziyad S, Tong F, et al. (2012). Subtype and pathway specific responses to anticancer compounds in breast cancer. Proc Natl Acad Sci US A109:2724-2729.

Comment 13: Ideally accuracy would be compared to existing breast cancer classifiers (e.g., using code from Marchionni et al., BMC Genomics, 2013) and/or survival curves reported in the literature.

Response: The proposed method has been compared against the K-TSP (Marchionni et al., BMC Genomics, 2013) as per reviewer’s suggestion and the results are presented in Table 6 of the manuscript.

Conclusions
Comment 14: Must be discussed in the context of existing genomics classifiers for breast cancer (e.g., OncotypeDx and/or Mammaprint).

Response: We have added text to both the second paragraph of the “Results and Discussion” paragraph and to the conclusion of the paper.

Comment 15: Results must be put in context with other predictions on METABRIC data, e.g., outcomes from the DREAM contest.

Response: An important distinction to note in regards to our methodology is that the predictions are based on the genes known to be associated with the response to specific drugs used to treat breast cancer. In the DREAM contest, the method with the highest METABRIC score (as described in Cheng et al., 2013) was phenotype-based, finding signatures for molecular processes that are disregulated in METABRIC, rather than responses to the cancer therapies themselves. While this is an interesting prediction method, the results cannot compared to our approach. The gene signatures that we have derived contain components of many different pathways.
Reference: Cheng WY, Ou Yang TH, Anastassiou D. Biomolecular events in cancer revealed by attractor metagenes. PLoS Comput Biol. 2013;9(2):e1002920.

View more View less

Competing Interests

PKR cofounded Cytognomix. A patent application related to biologically inspired gene signatures is pending. The other authors declare that they have no competing interests.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

Click here to access the data.

Downloaded data do not display as expected? Download the data (199.43KB)

[1] 1. Dorman SN, Baranova K, Knoll JH, et al.: Genomic signatures for paclitaxel and gemcitabine resistance in breast cancer derived by machine learning. Mol Oncol. 2016; 10(1): 85–100. PubMed Abstract | Publisher Full Text

[2] 2. Daemen A, Griffith OL, Heiser LM, et al.: Modeling precision treatment of breast cancer. Genome Biol. 2013; 14(10): R110. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Shoemaker RH: The NCI60 human tumour cell line anticancer drug screen. Nat Rev Cancer. 2006; 6(10): 813–823. PubMed Abstract | Publisher Full Text

[4] 4. Park NI, Rogan PK, Tarnowski HE, et al.: Structural and genic characterization of stable genomic regions in breast cancer: Relevance to chemotherapy. Mol Oncol. 2012; 6(3): 347–59. PubMed Abstract | Publisher Full Text

[5] 5. Hatzis C, Pusztai L, Valero V, et al.: A genomic predictor of response and survival following taxane-anthracycline chemotherapy for invasive breast cancer. JAMA. 2011; 305(18): 1873–1881. PubMed Abstract | Publisher Full Text

[6] 6. Curtis C, Shah SP, Chin SF, et al.: The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature. 2012; 486(7403): 346–352. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. MATLAB and Statistics Toolbox Release 2014a. The MathWorks Inc., Natick, Massachusetts, United States.

[8] 8. Hall M, Frank E, Holmes G, et al.: The WEKA data mining software: an update. ACM SIGKDD explorations newsletter. 2009; 11(1): 10–18. Publisher Full Text

[9] 9. Ding C, Peng H: Minimum redundancy feature selection from microarray gene expression data. J Bioinform Comput Biol. 2005; 3(2): 185–205. PubMed Abstract | Publisher Full Text

[10] 10. Rezaeian I, Mucaki EJ, Baranova K, et al.: Dataset 1 in: Predicting Outcomes of Hormone and Chemotherapy in the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) Study by Machine Learning. F1000Research. 2016. Data Source

Predicting Outcomes of Hormone and Chemotherapy in the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) Study by Biochemically-inspired Machine Learning

Abstract

Keywords

Introduction

Methods

Figure 1. Biochemically-inspired SVM gene signature derivation workflow.

Figure 2. RF decision tree diagram depicts the therapy outcome prediction process of a given patient, using a RF consisting of k decision trees.

Results and discussion

Table 1. SVM gene expression signature performance on METABRIC patients.

Table 2. Results of applying RF to predict outcome of paclitaxel therapy.

Table 3. Results of mRMR feature selection for an SVM for predicting outcome of paclitaxel therapy.

Figure 3.

Data availability

Author contributions

Competing interests

Grant information

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

The problem

How to fix it

Competing Interests Policy

Stay Updated