Several previous studies that explored the efficiency of various optimization algorithms for MLP in different areas of application showed that the most efficient optimized MLP varies depending on the area of application, sample sizes, and evaluation metrics implemented in such studies. From the studies reviewed, the most common area of research is information technology ^{31– 34} followed by the medical sector. ^{35– 37} To extend the study by, ³⁸ who focused on the financial sector, the current study uses a financial dataset, but it includes the CMA-ES-MLP and GOA-MLP, which are compared to the basic MLP and GA-MLP, which were also included in the study by, ³⁸ but across different sample sizes rather than only one. From the studies reviewed, the sample sizes ranged from 400 to 8367, but only one sample was used per study. As such, the current study expands the scope of these studies by comparing the basic MLP and its optimized variates using different samples to determine the effect of sample size on the performance of these ML algorithms.

The literature shows that in all the studies, the optimized versions of the MLPs were selected as the best performers, and not the basic MLP, which was not optimized. This is evident from the studies conducted by ³⁸ in which the diversity-considered GA-MLP ensemble algorithm (DGAMLPE) outperformed the unoptimized basic MLP, ³⁵ in which DGAMLPE outperformed the basic MLP, and, ³¹ in which the GOA-MLP outperformed the basic MLP. “This implies that indeed the optimised variates of the MLP can improve the basic MLP, and it is also seen that optimisation algorithms give the MLP a competitive advantage over other ML classifiers such as Random Forest (RF), Extreme Gradient Boost (X-GBoost), Weighted Count of Errors and Correct (WCEC), and Deep Belief Network-Support Vector Machine (DBN-SVM), Logistic Regression (LR), K-Nearest Neighbors (K-NN), Decision Tree Classifier (DTC), Support Vector Machine (SVM), Random Forest Classifier (RFC), and Ensemble” ³⁵ :314). Considering these findings from the literature, the researchers were interested in optimization algorithms for the MLP in the current study. To extend the literature, the researchers included the CMA-ES-MLP in the competing models and explored the effect of sample size on these ML algorithms.

The most used optimized variates of the MLP from previous studies are GA-MLP, ^{16, 32, 34, 35, 37} followed by PSO-MLP, ^{16, 31, 32, 34, 39} and GOA-MLP, ^{31, 33, 39} but none of these studies included CMA-ES-MLP, which implies that the performance of CMA-ES-MLP against GA-MLP and GOA-MLP remains an area that requires further research. This study bridges this gap. It also appears that the most frequently used accuracy metric from the reviewed studies is classification accuracy, ^{16, 31– 39} followed by the F-measure. ^{33, 35– 38} The negative and positive predictive values appear to be the least used accuracy metrics. ^{31, 35} Other classification evaluation metrics used in previous studies included sensitivity/recall, specificity, and precision. Similarly, the current study also implemented the classification accuracy, precision, sensitivity/recall, specificity, F-measure, and execution time to compare optimized algorithms based on the popularity of these metrics in previous studies. Including a variety of comparison metrics in a single study assists in minimizing the model selection bias that may be experienced when very few similar metrics are used in the comparison and selection of the most efficient model.

The data used in this study is a secondary dataset on the direct marketing campaigns of a Portuguese banking institution. The dataset was obtained from the UCI Machine Learning Repository of the Center for Machine Learning and Intelligent Systems. The primary contributor to the data is. ²⁸ The dataset can be accessed at https://archive.ics.uci.edu/ml/datasets/Bank+Marketing. The dataset has a total of 4 521 observations, and 11 variables were selected for use as attributes (see Table 1) in this study to predict whether a client will subscribe to a term deposit following the marketing campaign. That is, the binary variable “has the client subscribed to a term deposit” from the dataset is used as a dependent variable (binary; 0 is no and 1 is yes).

To mimic different sample sizes which are needed to study the impact of sample size on the efficiency of the MLP optimisation algorithms, nine (9) random samples of different sizes (varying by 10%) were drawn with replacement from the 4521. Samples were randomly selected at 10% difference using stratified sampling, in which the dependent variable was used as the stratum to ensure that the samples maintained the distribution of the main dataset in the dependent variable. The following random sample sizes were created: 10% (n = 452), 20% (n = 904), 30% (n = 1356), 40% (n = 1808), 50% (n = 2261), 60% (n = 2713), 70% (n = 3165), 80% (n = 3617), 90% (n = 4069), and the entire dataset, which contained 100% of the observations (n = 4521). The variables described in Table 1 were used as independent variables or features.

All categorical features with at least three (3) classes from Table 1 were converted to dummy variables using the one-hot encoding technique, which converts classes of the categorical variable to a vector that contains 1 and 0, denoting the presence and absence of the feature, respectively, which led to an increase in the number of features used in the paper to 42. Previous studies that have been conducted that focused on the application and/or comparison of ML classifiers (including MLP and its variates) on the dataset chosen for this study are summarized in Table 2.

Table 2 shows that 2004 to date, several ML classifiers have been evaluated to predict the likelihood of a client to subscribe to a term loan following a direct marketing campaign by the bank using data from a Portuguese banking institution. In general, the table shows that the results vary depending on the setting such as the number of attributes, the number of observations in the data, and the number of training times to mention a few. Most of these studies included neural networks ^{29, 40– 43, 47} including the basic MLP and its variates such as Meta-Cost-MLP, Cost sensitive classifier-MLP, and the GA based ANN (GA-ANN). Although it appears in most previous studies, the basic neural networks classifier was only found to be the best performer when compared to LR and DT and SVM in the study by. ²⁸ However, whenever its modified variates were included in the comparison, these variates were found to be best performers against the basic MLP such as in the study by ²⁹ in which the Meta-Cost-MLP outperformed the basic MLP and other classifiers such as (J48, LL, DT, VFDR), and in the study by ⁴⁷ in which the GA-ANN outperformed the baseline ANN. These results show that making improves to the basic MLP can improve its performance, hence this paper extend literature around the enhancement of the neural networks (specifically the MLP) as done by some authors in Table 2, by comparing GA, GOA and CMA-ES optimisation algorithms for the MLP using the direct marketing data used in studies that are summarise in this table. It is evident from Table 2 that these optimisation algorithms have never been compared in a single study using the dataset that was used by the studies in Table 2.

1.4.1 Data balancing

The data in this study were split into 80% training data and 20% testing data, which is a commonly used train-to-testing data-splitting ratio. A Synthetic Minority Oversampling Technique (SMOTE) was used to balance the training samples ⁴⁸ defined SMOTE as one of the most used oversampling techniques to solve imbalanced data problems, and it aims to balance class distributions by randomly increasing minority class examples by replicating them ⁴⁸ explained that SMOTE uses linear interpolation to generate the virtual training records. These synthetic data were generated through a random selection of at least one k-nearest neighbor for each observation in the minority class. ⁴⁸ In this study, SMOTE was chosen because of its advantage in reducing the risk of overfitting and its wide application in many previous studies, such as. ^{48– 52}

From Figure 1, Y i is the point under consideration, Y i 1 to Y i 4 are the nearest neighbors, and w 1 to w 4 represent the synthetic data generated by the randomized interjection ⁵³ explained that synthetic samples are generated by considering the difference between the nearest neighbor and the feature vector ⁵³ further explained that the difference is multiplied by a random number between 1 and 0 and then added to the feature vector under consideration. Table 3 presents balanced training data from the original dataset.

Table 3 shows that the class of the dependent variable is balanced after using SMOTE specifically for the training data samples. In all the samples, equal numbers of unsubscribed participants and subscribed participants are observed.

1.4.2 Multilayer Perceptron (MLP)

Explained that MLP was invented in 1958 at the Cornell Aeronautical Laboratory by Frank Rosenblatt, funded by the Office of Naval Research in the United States. ⁵⁴ Further explained that although it was originally designed as a machine rather than a program, the perceptron was first implemented in IBM 704 as software before being implemented in specially designed hardware as the “Mark 1 perceptron.” In addition, ⁵⁵ explained that the purpose of this machine is image recognition; it has 400 photocells arranged in an array and randomly connected to the “neurons.” According to the author, electric motors update the weights during learning, and the weights are encoded in the potentiometers. The flexibility of the MLP has enabled its function in various activities. ⁵⁶ It has only been used for image recognition, ^{57, 58} speech recognition, ⁵⁹ and machine translation software. ⁶⁰ Currently, it can be used for text data, ^{61, 62} speech recognition, ⁵⁸ and other types of data. MLP can be fitted using various software, such as Waikato Environment for Knowledge Analysis 3.9 (WEKA), Statistical Package for the Social Sciences (SPSS), and Python. With the use of optimization algorithms, such as those being compared in this study, MLPs have become very useful, convenient, and easy to use.

The MLP consists of an input and an output layer with one or more hidden layers of non-linear activating nodes. ⁶³ Each node in one layer connects with a certain weight to every node in the following layer. ⁶³ In the input layer, the activations, which were defined by ⁶⁴ as the source of the MLP’s power, were determined using the following equation: b j = ∑ i = 0 D w ij ( 1 ) x i , (1)

The first layer involves M linear combinations of the d-dimensional input for i 1 , 2 , … , M and j = 1 , 2 , … , d , where w ij ( 1 ) are the weights for node j in layer 1 for incoming node I and (1) indicates that this is the first layer of the network. Each activation was then transformed by a non-linear activation function g.

In this study, tanh was used as the activation function for the hidden layer ⁶⁵ described the Tanh function as a smoother, zero-center function, with a range between -1 and 1. The Tanh function is defined by the following equation sourced from ⁶⁵ : f ( x ) = ( e x − e − x e x + e − x ) , (2) where x is an input to the neuron and e is Euler’s number.

A sigmoid function was used as the activation function for the output layer. ⁶⁵ defined the sigmoid as a non-linear activation used mostly in feedforward neutral networks. “It is a bounded differentiable real function, defined for real input values, with positive derivatives everywhere and some degree of smoothness” ( ⁶⁵ :5). The sigmoid activation function is given by the following relationship, sourced from ⁶⁵ : f ( x ) = 1 1 + exp ( − b j ) (3) where f ( x ) corresponds to the outputs of the basis functions and is interpreted as the output of the hidden units.

1.4.3 Covariance Matrix Adaptation Evolution Strategy (CMA-ES)

The Covariance Matrix Adaptation Evolution Strategy (CMA-ES) was developed by Hansen et al. in 2003. ⁶⁶ According to, ⁶⁷ the algorithm’s theoretical underpinnings include variable metrics, and the CMA-ES uses maximum-likelihood updates in conjunction with a stochastic variable-metric approach. In an algorithm that quickly converges to the global optimum across a wide class of functions, the covariance matrix maximizes likelihood while resembling an expectation-maximization algorithm ^{68, 69} explained that the CMA-ES algorithm has certain drawbacks, such as its performance becoming slow if the number of model parameters that need to be estimated is large. The approximation of gradients without assuming or requiring their existence is another flaw of this algorithm. CMA-ES is a plausible candidate for an effective parameter estimation algorithm, ⁷⁰ but it must be tested against other algorithms to ascertain its efficiency, particularly when the sample size is varied.

The CMA-ES samples from the multivariate normal distribution search rank the sampled points according to their fitness function values. The multivariate normal distribution can be calculated using the following equation obtained by ⁷¹ : x i ∼ N ( m k , σ k 2 C k ) , (4) ∼ m k + σ k × N ( 0 , C k ) , (5) where m k is the distribution average and recent favorite solution to the optimization problem, σ k is the step size, and C k is the symmetric and positive definite

The fitness function for the CMA-ES is defined as: f ( x ) = g ( x T Hx ) , (6) where Hx is the Hessian matrix of f ( x ) and x T is the transpose of x .

The mean distribution is then updated to a weighted average using the following equation: m new ← ∑ i = 1 μ w i x i : λ = m + ∑ i = 1 μ wi ( x i : λ − m ) , (7) where m new is the new distribution mean, μ is the number of parameters, λ is the population size, m is the mean vector, and w i is the recombination weight.

The isotropic evolution is then updated using the following equation: p σ ← ( 1 − C σ ) p σ + 1 − ( 1 − C σ ) 2 μ w C k 1 2 m k + 1 − m k σ k , (8) where p σ is the evolution path, ( 1 − C σ ) is the discount factor, 1 − ( 1 − C σ ) 2 is the complement for the discounted variance, and μ w C k 1 2 m k + 1 − m k σ k are distributed as N ( 0 , I ) under neutral selection. σ k + 1 = σ k × exp ( C σ d σ ( | | p σ | | E | | N ( 0 , I ) | | − 1 ) ) , (9) where C σ d σ ( | | p σ | | E | | N ( 0 , I ) | | − 1 ) is unbiased about 0 under unbiased selection. E | | N ( 0 , I ) | | = 2 ( n + 1 2 ) ( n 2 ) , (10) ≈ n ( 1 − 1 4 n + 1 21 n 2 ) , (11)

Update of the covariance matrix adopted is described as follows: p c ← ( 1 − C c ) p c + 1 ( 0 , α n ) ( | | p σ | | ) 1 − ( 1 − c c ) 2 μ w m k + 1 − m k σ k , (12)

The CME-ES is finally updated using: C k + 1 = ( 1 − c 1 − c μ + c s ) C k + c 1 p c p c T + c μ ∑ i = 1 μ w i x i : λ − m k σ k ( x i : λ − m k σ k ) T (13) where c s is the small variance loss, c 1 is the learning rate for updating the covariance matrix, and c μ is the learning rate for rank- μ for updating the covariance matrix.

1.4.4 Genetic Algorithm (GA)

Reference [73] proposed a learning machine called the Genetic Algorithm (GA), which paralleled the principles of evolution. Barricelli (1954) pointed out that the first computer simulation of evolution was created in 1954 at the Institute for Advanced Study in Princeton, New Jersey, thanks to the efforts of Barricelli ⁷³ found that GA has some limitations, such as repeated evaluation of the fitness function and difficulties in working with dynamic datasets; it tends to converge to a local optimum or even arbitrary points, instead of the global optimum of the problem. “A better solution is only in comparison to other solutions, and the stop criterion is not clear in every problem” ( ⁷³ :226). On the other hand, GA has been noticed to be a very efficient and effective technique for both optimisation and ML applications. ⁷⁴ Another advantage of GA is that it requires less information about the problem ^{75, 76} stated that GA can work very well on mixed (discrete and/or continuous) problems. “The GA can be applied in real world situations such as engineering design, to make the design cycle process fast and economical, and in robotics too, to create learning robots which will behave as humans and will do tasks like cooking and laundry” ( ⁷⁷ : 347).

The efficiency of GAs depends on mutation and crossover operators and their relationships. “To determine the most appropriate operators, different mutation and crossover operators are used and they are compared with each other since GA involves a process of complex interaction between its parameters” ⁷⁸ suggested that for the algorithm to perform best, the population size must range between 50 and 100 observations. In this study, we verified this recommendation by studying the effectiveness of GA in different sample sizes ⁷⁹ stated that the algorithm comprises four main steps: selection, reproduction, replacement, and termination. The steps are as follows:

1.4.4.1 Selection

Reference [80] explained that by choosing the reproduction of offspring, the primary goal of this phase is to identify the area with the highest likelihood of producing a solution to the problem that is superior to that of the previous generation. The authors add that the selection of individuals will then be arranged in pairs of two to enhance reproduction ⁷⁹ also explained that individuals will then pass on their genes to the next generation. “The GA uses the fitness proportionate selection technique to ensure that useful solutions are used for recombination” ( ⁷⁹ : 3). Fitness proportion selection is defined by the author as the most popular method of parent selection, where every individual can become a parent with a probability that is proportional to its fitness. “Fitter individuals have a higher chance of mating and propagating their features to the next generation. Therefore, such a selection strategy applies a selection pressure to the more fit individuals in the population, evolving better individuals over time”( ⁸⁰ : 16). The fitness proportionate selection can be calculated using the following equation adopted from ⁸⁰ : p i = f i ∑ j = 1 N f i (14) where f i denotes the fitness of individual i in the population, N denotes the number of individuals in the population, and p i denotes the probability.

1.4.4.2 Reproduction

Reference [80] explained that the algorithm applies variation operators to the parent population during the reproduction phase, creating a child population. This phase has four main operators, crossover, mutation, replacement, and termination, which are discussed below.

1.4.4.3 Crossover

According to, ⁸¹ the crossover operator swaps the genetic information of two parents to produce offspring ⁸¹ also explained that this is performed on parent pairs that are selected randomly to generate a child population of equal size to the parent population. For this study, a single-point crossover was considered. “Single point crossover works in such a way that a parent organism string is selected. All data beyond this point in the organism string were swapped between the two parent organisms. Strings are characterized by positional bias” ( ⁸¹ : 13).

1.4.4.4 Mutation

The mutation operator adds genetic information to the new child population. According to, ⁸² the operator achieves this by flipping some bits in the chromosome to solve the problem of local minima and enhance diversification. In the present study, a bit-flip mutation was considered. “Bit flip mutation works in such a way that it selects one or more random bits and flip them. This can only be done for binary encoded GA’s” ( ⁸² : 47).

1.4.4.5 Replacement

Reference [80] elucidated that the replacement operator acts as the final generational step to replace the old population with the new child population. In this study, a generational replacement operator is used, where the previous generation is replaced with a newly generated child population.

1.4.4.6 Termination

Reference [80] explains that termination is only possible in specific situations, such as having reached an absolute number of generations but not having improved the population for X iterations or the objective function value reaching a pre-defined threshold ⁷⁹ cited a genetic algorithm example in which a counter was maintained to record generations for which the population did not improve. “Initially, we set the counter to zero. Each time we do not generate an offspring, which is better than the individuals in the population, we increase the counter. However, if the fitness of any offspring is better, then we reset the counter to zero” ( ⁷⁹ : 2). The author also stated that the algorithm terminates when the counter reaches a predetermined value.

1.4.5 Grasshopper Optimisation Algorithm (GOA)

The Grasshopper Optimisation Algorithm (GOA) is a new swarm intelligence algorithm and population-based method developed by Seyedali Mirjalili in 2017. ⁸³ According to the authors, the GOA mainly observes the behavior of grasshopper swarms and their social interactions. Every grasshopper in the population represents a solution, and its location within the swarm is determined by three forces: wind advection, the force of gravity applied to it, and social interactions with other grasshoppers. ⁸⁴ The process of optimizing the grasshopper algorithm involves several steps, including initialization, creation, and evaluation of the first population, identification of the best overall solution, updating the decreasing coefficient parameter, mapping the grasshopper’s distance, and updating the solution. ⁸⁵

Reference [87] explained that the GOA can improve the average fitness of all grasshoppers, which helps the GOA effectively increase the first randomly generated solutions. The algorithm can be computed using software such as Matrix Laboratory (MATLAB) and Python. No information relating to the GOA in comparison with other algorithms has emerged, as this is a newly developed algorithm. Therefore, little is known about the efficiency of this algorithm compared to its predecessors; hence, the proposed study seeks to expand the scope of this algorithm. Grasshopper position ( X i ) calculations depend on three types of forces: social interactions and other grasshoppers, wind advection, and gravitational force. ⁸⁷ All equations used in the description of the GOA in this study were sourced from ⁸⁷ the grasshopper’s position is defined as: X i = S i + G i + A i , (15) where X i defines the position of the i-th grasshopper, S i is the social interaction, G i is the gravitational force on the i-th grasshopper, and A i is wind advection.

From Equation 15, social interaction is defined as: S i = ∑ J = 1 N s ( d ij ) d ij , ̂ (16) where d _ij is the distance between grasshopper i and grasshopper j in the d th dimension.

From Equation 15, the gravitational force ( G i ) on the grasshopper is computed as follows: G i = − g e g ̂ , (17) where − g denotes the gravitational constant and e ̂ g is the unit vector towards the center of the earth.

From Equation 15, the wind advection is computed as follows: A i = ue ̂ w , (18) where u is a constant drift and e ̂ g represents a unity vector towards the direction of the wind.

When substituting Equations 16– 18 into Equation 15, the position of the current grasshopper becomes. X i = ∑ j = 1 j ≠ 1 N s ( | x j − x i | ) x j − x i d ij − ge ̂ g + ue ̂ w (19) where N is the total number of grasshoppers.

Reference [84] explained how the pseudocode of the GOA algorithm works. The GOA starts optimization by creating a set of random solutions; the search agents then update their positions, followed by the determination of the position of the best target obtained thus far, and this position is updated in each iteration. ⁸³ Additionally, the distances between grasshoppers were normalized in each iteration ⁸³ stated that position updating is performed iteratively until the end criterion is satisfied. Finally, the position and fitness of the best target are returned as the best approximation of the global optimum.

1.4.6 Model comparison criteria

Precision, sensitivity/recall, F-score, classification accuracy, sensitivity, specificity, and execution time were used to evaluate and compare the optimization algorithms for the MLP, as described in this section. The classifier with the highest precision, recall, F-score, accuracy rate, sensitivity, specificity, and lowest execution time is preferred.

Classification accuracy (also referred to as overall accuracy) was described by ⁸⁸ as the number of correct forecasts divided by the total number of forecasts. It is the most straightforward clustering quality measure proposed by ⁸⁹ to assess the clustering results related to the ground truth. ⁸⁸ Classification accuracy was calculated by ⁸⁸ as follows: Accuracy = True Positives + True Negatives ( Positives + Negative ) (20)

Reference [91] characterized specificity as a proportion of the extent of real negatives that are effectively distinguished, and they described the specificity equation as follows: specificity = True Negatives True Negatives + False Positives (21)

Precision was defined by ⁹¹ as a measure of how close a series of measurements are to one another. The author explained that precise measurements are highly reproducible, even if the measurements are not near the correct value. Precision was calculated as follows ⁹¹ : Precision = True Positives True Positives + False Positives (22)

Reference [91] characterize the sensitivity/recall rate as a measure of the proportion of real positives that are accurately identified. The following equation for recall/sensitivity was adopted from ⁹⁰ : Sensitivity / recall = True Positives True Positives + False Negatives (23)

Reference [93] defined the F-measure as a weighted harmonic mean of recall and precision. There are several motivations for this choice ⁹² explains that the harmonic mean is commonly appropriate when averaging rates or frequencies, but there are also a set of theoretical reasons. The author further explains that the mean allows differential weighting of recall and precision, but they are commonly given equal weights. The F-measure was computed as follows: F = 2 ∗ Precision ∗ Recall Precision + Recall (24)

Execution time is defined by ⁹³ as the amount of time spent by the system executing a given task, including the amount of time it spends executing runtime or system services.