The rise and fall of machine learning methods in biomedical research

In the era of explosion in biological data, machine learning techniques are becoming more popular in life sciences, including biology and medicine. This research note examines the rise and fall of the most commonly used machine learning techniques in life sciences over the past three decades.

The complexity of biological data has been growing even faster, and doesn't seem to be linearly dependent on the size of data. Examples of complexity in the field of computational genomics include multiple diverse sources of technical noise, low signal to noise ratio, low numbers of biological replicates in comparative approaches, rare and usually hardly detectable mutations in non-coding regions and rare and barely identifiable cell types in complex heterogeneous systems such as the immune system and/or the brain.
At the intersection of mathematics, statistics and computer science is machine learning (ML), the de facto tool box in data science for deciphering the relationship between the input and output as well as detecting significant patterns within large, complex data sets. These quantitative approaches have been shown to be effective and are becoming increasingly popular in addressing challenges such as those outlined above. Highlights of their successful applications in functional genomics include, but are not limited to, learning and characterizing chromatin states by employing unsupervised approaches such as chromHMM (Ernst & Kellis, 2012), predicting sequence specificities of DNA-and RNA-binding proteins using convolutional neural networks such as DeepBind (Alipanahi et al., 2015), and employing a combination of supervised and unsupervised approach to determine the genetic and epigenetic contributors of antibody repertoire diversity (Bolland et al., 2016). Nowadays it is almost impossible to publish a study on single cell assays without using dimensionality reduction methods such as Principal Component Analysis or t-SNE.
One indirect measure of the success of these techniques in extracting scientific insights from biological data is to measure the popularity and usage of machine learning algorithms in life sciences research over time (Jensen & Bateman, 2011). Motivated by Jensen et al., I therefore set out to update machine learning usage in life sciences. For this I quantified what fraction of published papers in the PubMed database mention a particular technique and how these number of citations are changed each year (see methods).

Methods
For this analysis, I used the R RISmed package (Kovalchik, 2015) to parse the publication data from NCBI. I examined publications in PubMed from 1990 to 2017 using a metric that measures the proportion of publications per year that mention the technique in the full text (Hits Per Year per Million articles published, or HPYM). The Popularity Rate (PR) of a technique was then defined as the difference between HPYMs in any two consecutive years. A positive PR shows an increase in popularity, whereas a negative PR reflects a decrease in popularity. I limited this note to 12 models listed in Table 1 which have been the most common or which showed a sharp change in popularity rate at a particular time. However, the R code is available with which any particular model during a specific period of time can be easily measured.

Results
This analysis demonstrates that the overall popularity of machine learning methods in biomedical research has linearly increased since 1990 to 2017, but with two different slopes. From 1990 to 2000 the slope is 0.02, meaning that popularity increased only 2% per year. In 2001 (when sequencing big genomes became possible) the slope increased to 0.06, and since then it has remained constant. A maximum of 1.2% of all papers published in PubMed in any calendar year have mentioned one of the machine learning methods investigated in this study

Amendments from Version 1
In this new version, I have tried to address all the comments and suggestions raised by both of my referees. I have therefore changed the manuscript's contents, structure and figures accordingly. Therefore, the new version has now three main figures whereas the previous version had only one figure. I have additionally made some changes to the R code and updated the online git project.
To address R1's key comments, earlier in the manuscript, I have acknowledged that this work has been motivated by a previous similar publication. I have made the distinction between ANNs and DNNs clearer throughout the manuscript. All minor points by R1 has been also addressed.
To address R2's key concerns: I have added proper referencing to the R code to prevent from the error.
Both 'naïve bayes classifier' and 'logistic regression' models have been re-added to the manuscript and to the figures.
The issue of counting hits for DNN has been resolved and manuscript has been updated accordingly.
Figures have been re-structured and now I have three figures instead of only one. The PR has been illustrated in one separate figure.
I have also tried to address all the possible minor points.
I have added a full detailed response to each of the referees in online 'Respond or Comment' section of the manuscript.

Table 1. Common Machine Learning Techniques in Life
Sciences. This table shows 12 machine learning techniques whose popularity in life sciences have been investigated in this study. Technical note: Supervised means that the model requires training data to learn its parameters. A supervised model is used to predict the future instances. An unsupervised model doesn't require any training data and is used to detect patterns within a dataset. Dimensionality reduction models are used to project high-dimensional datasets into lower dimension space where new variables are more interpretable.

Technique
Abbreviation Category  Figure 1). I was expecting to see a higher usage of ML in life sciences, but without a gold standard set to compare with, I would not be able to judge if this is too high or low or just about right.

Random
The Linear Regression (LR) models have been the most dominant machine learning techniques in the life sciences over the past three decades (Figure 2A). It is interesting to see that LR models are still highly in used despite recent appearance of sophisticated ML techniques such as ensemble-based approaches and/or Support Vector Machines and even with very recent and state of the art deep learning techniques. Although, its popularity rate has been plateaued over the past few years ( Figure 3) meaning that its usage is increased linearly with a constant slope. With a constant increase of 300 HPYM, and considering its higher intercept at 1990, the linear regression models is predicted to be one of the most popular techniques over the next few years.
Perhaps a very surprising observation of this study is the rise and fall of Principle Component Analysis (PCA). PCA became very fashionable during 2000 to 2013. In fact, 3329 per million papers published in 2013 mentioned PCA which was the highest number of PCA usage. Since then it has been used less, although it still is the second most popular tool ( Figure 2A).
In early 2000s, unsupervised Hierarchical Clustering alongside newly introduced supervised techniques Support Vector Machines (SVMs) and Random Forests (RFs), showed a sharp rise in  usage, which was mainly associated to microarray data analysis. Usage of hierarchical clustering plateaued shortly after its sharp popularity rise in 2000. SVMs kept their popularity longer, for almost a decade in fact, but subsequently dropped to an almost negligible popularity rate ( Figure 3). RFs on the other hand, showed less popularity at the beginning of their arrival, but later on (after 2013) they were ranked the second highest in popularity after Deep Neural Networks (DNN) (Figure 2A, 2B and Figure 3).
During the period of 1990-2017, Artificial Neural Networks (ANNs) have demonstrated considerable fluctuations in popularity ( Figure 2B and Figure 3). ANNs in the early 1990's after Linear Regression and PCA, were the most commonly used techniques until early 2000, when they lost their popularity to MMs, HCs and SVMs and even later to RFs. However, since 2013, a subfamily of ANN known as Deep Neural Networks (DNNs) made their way into the life sciences, and their usage since then has increased remarkably, so that DNNs currently have the highest popularity rate ( Figure 3).
The dimensionality reduction technique t-distributed Stochastic Neighbour Embedding (t-SNE) published in 2008, has become quickly tailored to all sorts of single cell techniques. It is therefore not surprising to see that t-SNE usage has also been very rapidly growing over the past few years ( Figure 2B). Discussion I have illustrated the rise and fall of ML techniques in life sciences from 1990 to the present day. I chose this period because I believe this is the transition period for life scientists to join the big-data club. With the same R code used in this study to parse the publication data from NCBI, it would be possible to look at any period of time.
It was not very surprising to see LR models as the most commonly used model in the field, since: a) LR models are one of the oldest ML methods that have been in use in almost any field, b) Parameters in LR models can be learned by using a training data with just a few data samples.
c) A lot of other models can be placed under this umbrella, for instance by first applying a transformation function.
It was, however, surprising to see the sharp rise and fall of PCA. Perhaps a contributing factor to PCA being the most dominant dimensionality reduction method available in this period was its easy-to-use implementation in R. The question still remains as to why its popularity decreased from 2008 onwards.
Perhaps the arrival of more versatile models such as RFs and SVMs which are very capable of handling high dimensionality and dealing with co-linearity in biological data eased the need to use PCA. Additionally, t-SNE as a tremendously growing dimensional reduction model in the field, is establishing itself as a strong competitor for the PCA.
ANNs have been fairly popular since the 1990s until around 2004. Around that time more readily useable and less complex techniques became available, such as SVMs, RFs and MMs. However, with the huge investments of giant information companies such as Google leading to very impressive applications of DNNs and other sub-families of ANNs, in various disciplines, DNNs, currently has the sharpest popularity rate (Figure 3).
I appreciate that there are limitations to this study. For instance, for the majority of comparative analyses of gene expression, researchers use a differential expression software and/or package, but cite only the package name and not the underlying statistical or ML technique used in the package. These cases have not been covered in this study. However, this study can be considered as an approximation of the extent to which machine learning techniques are used in life sciences. In the manuscript "The rise and fall of machine learning methods in biomedical research" the author has generated a quantitative perspective on the usage of machine learning methods in the life sciences. For some of the methods a hypothesis about the underlying reason for an increased or decrease popularity are discussed. The code for performing the analysis is available on GitHub and -like the retrieved PubMed data -has been deposited at Zenodo. I have several major objections / question / suggestion for the author: I tried to reproduce the analysis using RStudio 1.1.383 with the deposited RStudio project but got the following error when executing the R chunks in the file : "Error Machine_Learning_Trends.Rmd in library(informationRetrieval) : there is no package called 'informationRetrieval'" The file is located in another subfolder and I guess this just needs proper informationRetrieval.R referencing inside of the project. The author states that he has selected widely used machine learning methods used in life sciences. I would have expected Naive Bayes classifiers in the list of most popular methods. A simple PubMed search for '"naive bayes classifier" OR "naive bayesian classifier' return twice as simple PubMed search for '"naive bayes classifier" OR "naive bayesian classifier' return twice as many hits as for "deep neural networks" (but over a longer time span): https://www.ncbi.nlm.nih.gov/pubmed/?term=%22naive+bayes+classifier%22+OR+%22naive+bayesian+ https://www.ncbi.nlm.nih.gov/pubmed?term=%22deep+neural+networks%22 Similar issue for logistic regression: The analysis in the provided file actually contains the counting of publications containing logistic Machine_Learning_Trends.Rmd regression that shows a large (206,619 at the time of writing) and growing number of this but this method has not been discussed in the manuscript and is not displayed in the plots.
https://www.ncbi.nlm.nih.gov/pubmed?term=%22logistic%20regression%22 The counting of hits for deep neural networks (DNN) is not done properly. Looking at the code to count the number of hits of different search terms shows that the author use "artificial neural networks" and "deep neural networks" and "deep learning" as search term for DNN (see code selection at the bottom of this section). I think using the search term "artificial neural network" for both ANN and DNN is not sound and changes the story of DNN (a special form ANN) significantly. Either DNN is treated as subset of ANN and only ANN are plotted or DNN and ANN are treated separately and the search term "artificial neural network" is not used for DNN. Furthermore the search term "deep learning" results in numerous unrelated hits before 2010 (e.g. PMID: 8936230, 9165817, 9487168, 10463930).

NN_term <-"(artificial neural networks[tw] OR deep neural networks[tw] or deep learning[tw])"
DNN_hits <-get_normliazed_number_of_hits(years=YEARs, query=NN_term, db="pubmed", normalization_value=1000000)`M inor issues: Figure 1 Style: The different lines are hard to distinguish by color only -maybe consider an additional Style: The different lines are hard to distinguish by color only -maybe consider an additional discriminator (e.g. dashed lines for a subset); Next to Fig 1 C is a lot of white space. Placing the t-SNE subplot to a different location (e.g. the middle of Fig 1 C) would make it possible to use this space more efficiently. Maybe think rearranging the whole figure. Figure 1 C is a subplot of figure 1 B like the t-SNE plot is a subplot of Figure 1 C "de facto" should be written in in italic font The link to RISmed should use the link indicated at the page itself that says "Please use the canonical form to link to this page." https://CRAN.R-project.org/package=RISmed For Linear Regression Model sometimes "LRM" and sometime "LR" is used in the manuscript In order to understand which biological approaches / questions that are influencing the usage of different ML method the association of those methods with certain MeSH term would be interesting. Either as part of this manuscript or a future one.

If applicable, is the statistical analysis and its interpretation appropriate? Partly
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Partly
No competing interests were disclosed.

Competing Interests:
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 22 Dec 2017 , Oxford University Hashem Koohy I thank Dr Konrad Forstner for his time in evaluating the manuscript and for his very detailed comments/suggestions that I believe will immensely enhance the quality of the manuscript.
In the following I address the issues raised by Konrad.
In the manuscript "The rise and fall of machine learning methods in biomedical research" the author has generated a quantitative perspective on the usage of machine learning methods in the life sciences. For some of the methods a hypothesis about the underlying reason for an increased life sciences. For some of the methods a hypothesis about the underlying reason for an increased or decrease popularity are discussed. The code for performing the analysis is available on GitHub and -like the retrieved PubMed data -has been deposited at Zenodo. I have several major objections / question / suggestion for the author: I tried to reproduce the analysis using RStudio 1.1.383 with the deposited RStudio project but got the following error when executing the R chunks in the file : "Error in library(informationRetrieval) : there is no package Machine_Learning_Trends.Rmd called 'informationRetrieval'" The file is located in another subfolder informationRetrieval.R and I guess this just needs proper referencing inside of the project. I have changed the package structure and added a cell on top of the rdm file so that it should easily find the *.r codes and source them.
The author states that he has selected widely used machine learning methods used in life sciences. I would have expected Naive Bayes classifiers in the list of most popular methods. A simple PubMed search for '"naive bayes classifier" OR "naive bayesian classifier' return twice as many hits as for "deep neural networks" (but over a longer time span): https://www.ncbi.nlm.nih.gov/pubmed/?term=%22naive+bayes+classifier%22+OR+%22naive+bayesi https://www.ncbi.nlm.nih.gov/pubmed?term=%22deep+neural+networks%22 In my very initial analysis I had naïve bayes classifier, but to my surprise after normalization it was overshadowed by other highly used techniques. I therefore took it out. Now, for the completion, I have added it again.
Similar issue for logistic regression: The analysis in the provided file actually contains the counting of publications containing Machine_Learning_Trends.Rmd logistic regression that shows a large (206,619 at the time of writing) and growing number of this but this method has not been discussed in the manuscript and is not displayed in the plots. Similarly, I had logistic regression models in my initial analysis. And for the same reason I took it out from the final submission and left to the reader to test if they wish. It has been added again. https://www.ncbi.nlm.nih.gov/pubmed?term=%22logistic%20regression%22 The counting of hits for deep neural networks (DNN) is not done properly. Looking at the code to count the number of hits of different search terms shows that the author use "artificial neural networks" and "deep neural networks" and "deep learning" as search term for DNN (see code selection at the bottom of this section). I think using the search term "artificial neural network" for both ANN and DNN is not sound and changes the story of DNN (a special form ANN) significantly. Either DNN is treated as subset of ANN and only ANN are plotted or DNN and ANN are treated separately and the search term "artificial neural network" is not used for DNN. Furthermore the search term "deep learning" results in numerous unrelated hits before 2010 (e.g. PMID: 8936230, 9165817, 9487168, 10463930).
(then click on the https://www.ncbi.nlm.nih.gov/pubmed/?term=%22deep+learning%22 "Result by year" histogram). I really apologize for this. I have corrected in the code, and changed the manuscript accordingly.
The authors tries to explain the underlying reasons for the gain or loss of certain ML methods. In Figure 1  The manuscript now has three main figures which are hopefully more clearer than the previous version. "de facto" should be written in in italic font Corrected. The link to RISmed should use the link indicated at the page itself that says "Please use the canonical form to link to this page." https://CRAN.R-project.org/package=RISmed For Linear Regression Model sometimes "LRM" and sometime "LR" is used in the manuscript I have corrected for this.
In order to understand which biological approaches / questions that are influencing the In order to understand which biological approaches / questions that are influencing the usage of different ML method the association of those methods with certain MeSH term would be interesting. Either as part of this manuscript or a future one. This is a very interesting point. Though as suggested is beyond this manuscript. I should firstly point out that I was co-author on the 2011 editorial published in Bioinformatics titled, "The rise and fall of supervised machine learning techniques" . Therefore I was momentarily surprised to be invited to review a paper with such a similar title. That editorial was only a page and a half long and only really scratched the surface of the interesting topic of the prevalence of use of machine learning in the biosciences. The author cites our 2011 paper and mentions that the current article can be considered an update of it. However, that is only mentioned in the very final paragraph of the paper. It would seem reasonable to me to make that one of the first things mentioned in the paper. Of course I am far from a neutral observer on this point.
Overall I think that the article presents sound and interesting science and should be published within F1000Research. I think it provides a timely update to the 2011 editorial and expands it with some nice extra details. The article increases the number of ML methods investigated from 5 to 10. Most notably linear regression models are included which top the league table.
I noticed an inconsistency in the data presented for ANNs in this new paper compared to the 2011 paper. Why is that? The numbers for ANN are considerably lower in this article. Is that because DNNs are split out from ANNs? Throughout the paper it says that ANNs have become known as DNNs. That is not correct. DNNs are a subtype of ANNs. So all DNNs are ANNs, but not all ANNs are DNNs. That needs correction throughout.
The following statement does not read well: "The sharp increase usage in popularity rate of DNNs over the past few years ( Figure 1C) suggests that DNNs will take the PR lead again in the coming years." After multiple readings I would presume that PR lead means it has the highest popularity rate. DNNs would have more than 300 more mentions per million papers per year. Firstly that sentence is very confusing to understand for a reader. For the first two readings I thought you were saying that DNNs would take the lead from LRMs, which would seem unlikely. On third reading I thought you meant that the slope of DNNs would overtake LRMs, but clearly it has already done that. I think you should rethink that sentence or take it out.

Minor points:
Page 2. At he intersection -> At the intersection Page 2. You mention that a surprising maximum of 1.2% of all paper mention one of the 10 ML 1