Stacked deep analytic model for human activity recognition on a UCI HAR database

Background Owing to low cost and ubiquity, human activity recognition using smartphones is emerging as a trendy mobile application in diverse appliances such as assisted living, healthcare monitoring, etc. Analysing this one-dimensional time-series signal is rather challenging due to its spatial and temporal variances. Numerous deep neural networks (DNNs) are conducted to unveil deep features of complex real-world data. However, the drawback of DNNs is the un-interpretation of the network's internal logic to achieve the output. Furthermore, a huge training sample size (i.e. millions of samples) is required to ensure great performance. Methods In this work, a simpler yet effective stacked deep network, known as Stacked Discriminant Feature Learning (SDFL), is proposed to analyse inertial motion data for activity recognition. Contrary to DNNs, this deep model extracts rich features without the prerequisite of a gigantic training sample set and tenuous hyper-parameter tuning. SDFL is a stacking deep network with multiple learning modules, appearing in a serialized layout for multi-level feature learning from shallow to deeper features. In each learning module, Rayleigh coefficient optimized learning is accomplished to extort discriminant features. A subject-independent protocol is implemented where the system model (trained by data from a group of users) is used to recognize data from another group of users. Results Empirical results demonstrate that SDFL surpasses state-of-the-art methods, including DNNs like Convolutional Neural Network, Deep Belief Network, etc., with ~97% accuracy from the UCI HAR database with thousands of training samples. Additionally, the model training time of SDFL is merely a few minutes, compared with DNNs, which require hours for model training. Conclusions The supremacy of SDFL is corroborated in analysing motion data for human activity recognition requiring no GPU but only a CPU with a fast- learning rate.


Introduction
Human activity recognition (HAR) can be categorized into vision-based and sensor-based.In vision-based HAR, an image sequence, in the form of video, recording the human activity is captured by a camera. 1 This sequence will be analysed to recognize the nature of an action.This system is applied for surveillance, human-computer interaction and healthcare monitoring.For sensor-based HAR, human activities are captured by inertial sensors, such as accelerometers, gyroscopes or magnetometers.Among these approaches, sensors are more favourable due to their lightweight nature, portability and low energy usage. 2 With the advancement of mobile technology, smartphones are equipped with high-end components.Accelerometer and gyroscope sensors embedded in the smartphone make it feasible as an acquisition device for HAR.4][5][6] In this work, we categorize the smartphone-based HAR as part of the sensor-based HAR.Activity inertial signals are collected through smartphone sensors.

Related work
Hand-crafted approaches using manually computed statistical features have been proposed. 4,7These authors applied various machine learning techniques such as decision tree, logistic regression, multilayer perceptron, naïve Bayes, Support Vector Machine etc. to classify the detected activities.The performance of the handcrafted approaches might be affected when dealing with complex scenarios due to their feature representation incapability.The algorithms could easily plummet into the local minimum despite the global optimal.
Hence, various deep neural networks (DNNs) were explored in HAR owing to the capability of extracting informative features.DNN is a machine learner that can automatically unearth the data characteristics hierarchically from lower to deeper levels. 8,9The work of Ronao and Cho (2016), 10 Lee et al. (2017) 11 and Ignatov (2018) 12 explored the deep convolutional neural networks by exploiting the activity characteristics in the one-dimensional time-series signals captured by the smartphone inertial sensors.The empirical results substantiated that the extracted deep features were crucial for data representation with promising recognition performance.Zeng et al. (2014) proposed a modified convolutional neural network to extract scale-invariant characteristics and local dependency of the acceleration time-series signal. 13The weight sharing mechanism in the convolutional layer was modified.Unlike in the vanilla model where the local filter weights were shared by all positions within the input space, the authors incorporated a more relax weight sharing strategy (partial weight sharing) to enhance the performance.
Recurrent Neural Network (RNN) was proposed to process sequential data by analysing previously inputted data and processing it linearly.Due to the vanishing gradient problem, RNN was enhanced and Long Short-Term Memory -LSTM was introduced.Chen et al. (2016) explored the feasibility of LSTM in predicting human activities. 14Empirical results demonstrated an encouraging performance of LSTM in HAR.Further, an enhanced version of LSTM, i.e. bidirectional LSTM, was proposed. 15Unlike LSTM, bidirectional LSTM tackles both past and future information during the feature analysis.With this, a richer description of features could be extracted for classification.
A cascade ensemble learning (CELearning) model was proposed for smartphone-based HAR. 16There are multiple layers in this aggregation network and the model goes deeper layer by layer.Each layer contains Extremely Gradient Boosting Trees, Random Forest, Extremely Randomized Trees and Softmax Regression.The CELearning model gains higher performance, and the training process is rather simple and efficient.Besides, Hierarchical Multi-View Aggregation Network (HMVAN) is also one of the aggregation models. 17This model integrates features from various feature spaces in a hierarchical context.In this network, three aggregation modules from the aspect of feature, position and modality levels are designed.Further, a modified Dynamic Time Warping (DTW) has been proposed for template selection in human activity recognition. 18Empirical results showed that the modified DTW was able to improve on the computational efficiency and similarity measure accuracy.In viewing the significance of time-series and continuous characteristics of sensor data, a two-stage continuous hidden Markov model framework was proposed by taking advantage of the innate hierarchical structure of basic activities. 19This framework could diminish feature computation overhead by manipulating different feature subsets on different subclasses.Experiments showed that this proposed hierarchical structure drastically boosted the recognition performance.

REVISED Amendments from Version 2
Correction on the mathematical definitions/equations to rectify the inconsistent variables used in the definitions/equations.Any further responses from the reviewers can be found at the end of the article

Motivation and contributions
In DNNs, there are learning modular components in multiple processing layers for multiple-level feature abstraction.These layers are trained based on a versatile learning principle, no requiring any manual design by experts. 20DNNs accomplish excellent performances.However, these networks are not well trained if they have limited training samples, leading to performance degradation.Furthermore, there is a lack of theoretical ground on how to fine-tune the gigantic hyper-parameter series. 17The outstanding accomplishment of DNNs can only be achieved if and only if sufficient training data is accessible for fine-tuning the numerous parameters.A high specification of GPU is needed to train the network from gargantuan datasets.Besides, impersonal HAR solution is preferable for real-time applications.This solution can be directly applied to new users without necessitating model regeneration.
A stacking-based deep learning model for smartphone-based HAR is proposed.Inspired by the hierarchical learning in the DNNs, the proposed stacked learning network is aggregated with multiple learning modules, one after another, in a hierarchical framework.Specifically, a discriminant learning function is implemented in each module for discriminant mapping to generate discriminative features, level by level.The lower (generic) to deeper features are input to a classifier for activity identification.This proposed approach is termed Stacked Discriminant Feature Learning (SDLF).
The contributions of this work are summarized: 1.A deep analytic model is proposed for smartphone based HAR for quality feature extraction without the need of a gigantic training set and tenuous hyper-parameter tuning.
2. An adaptable modular model is developed with a discriminant learning function in each module to extract discriminant deep features demanding no graphics processing unit (GPU) but only a central processing unit (CPU) with a fast-learning rate.
3. An experimental analysis using various performance evaluation metrics (i.e.recall, precision, the area under the curve, computational time, etc.) with subject-independent protocol implementation (no overlap in subjects between training and testing sets) to facilitate impersonal HAR solution.

Methods
Smartphone inertial sensors were used to capture 3-axial linear (total) acceleration and 3-axial angular velocity signals.
These signals were pre-processed into time-and frequency-domain features, as listed in Table 1.Next, the pre-processed data was inputted into the Stacked Discriminant Feature Learning (SDFL) for feature learning.The extracted feature template was fed into the nearest-neighbour (NN) classifier for classification.The overview of the system is illustrated in Figure 1.
SDFL is a pile of multiple discriminant learning layers interleaved with a nonlinear activation unit, as illustrated in Figure 2. By cascading multiple discriminant learning modules, each layer of SDFL learns based on the input data and the learned nonlinear features of the preceding module.The depth of the stacking layer is determined using the database subset.If the performance is not improving but showing degradation, the depth of the stacking layer is determined.In this case, the depth of three showed the optimal performance, so we adopted this architecture with three layers.To be detailed, the first discriminant learning module learns based on the input data and the second learning module learns based on an input vector (concatenating the input data and the learned features of the first learning module).This is similar to the third learning process where the third learning module learns based on an input vector (comprising the input data and the learned features of the second learning module).
g N i¼1 be a set of N transformed data with d dimension, i.e. d = 561, y i is the class label of x i , C is the number of training classes, i.e.C = 6, each of C classes has a mean μ j and total mean vector μ ¼ 1 C P C j¼1 m j μ j with m j denotes the number of training samples for jth class.In the first learning layer, the input vector is the transformed data x i .The computation of the intrapersonal scatter matrix Σ intra and interpersonal scatter matrix Σ inter are defined as:   where T denotes a transpose operation.Next, a linear transformation Φ is computed by maximizing the Rayleigh coefficient.With this optimization, the data from the same person could be projected close to each other, while data from different people is projected as far apart as possible.This optimization function is termed as Fisher's criterion, 21 The mapping Φ is constructed through solving the generalized eigenvalue problem, The learned features are produced through the projection of the input data x i onto the mapping subspace, x is transformed to C À 1 dimensions.We denote l for the index of modular layer in SDFL.The learned feature vector of the first modular unit is notated as xi activation function.In this study, sigmoid function, 1þe Àx i is used for the nonlinear projection.To be specific, ð Þ is the nonlinear learned features of the first modular unit.
For deeper modules, the input vector of the respective module is a stacking vector containing the input data and the learned features, i.e.
where l ¼ 2 and 3.The intrapersonal scatter matrix In this case, μ j l ð Þ is the jth class mean computed from the input vectors of jth class, z i l ð Þ ∈C j and the total mean vector at lth modular unit.The final feature vector is the nonlinear learned features of each modular layer with length of 3(C-1), In this work, the experimental hardware platform was constructed on a desktop with an Intel ® Core™ i7-7700 processor with 4.20 GHz and 48.0 GB main memory; whereas the experimental software platform was a 64-bit operating system of Windows 10 with Matlab R2018a (MATLAB, RRID:SCR_001622) software (An open-access alternative that provides an equivalent function is GNU Octave (GNU Octave, RRID:SCR_014398)).We used the UCI HAR dataset 7 : There were 30 subjects with 7352 training samples and 2947 testing samples.Each subject was required to carry a smartphone (Samsung Galaxy SII) on the waist and perform six activities: walking, walking_upstairs, walking_downstairs, sitting, standing and laying.

Results
We scrutinized how well SDFL could analyse the inertial data and correctly classify those activities in a user-independent scenario.The challenge of subject-independent protocol is that there is a certain degree of variance of human gait towards the inertial data patterns, although performing the same activity, as illustrated in Figure 3 (for standing).
In this work, UCI HAR dataset was partitioned into two sets: 70% of the volunteers were selected to generate the training data and the remaining 30% of the volunteers' data was used as the testing data.There was no subject overlapping between the training and test data sets.Table 2 records the performance of SDFL and Figure 4 shows the confusion matrix.The performance comparison with other approaches is recorded in Table 3.The computational time of SDFL is tabulated in Table 4.

Discussion
From the empirical results, we observed that the proposed SDFL was able to demonstrate superior classification performance compared to most of the existing techniques, even though a simple classifier was adopted in the system.The exceptional performance of SDFL explains the capability of SDFL in capturing the essence of the inertial data without heavily depending on the classifier.Furthermore, SDFL also exhibited its superiority to most of the existing approaches, including deep learning models.To be specific, SDFL obtained an accuracy of 96.3%, whilst Deep Belief Network's accuracy was 95.8%, 9 CNN achieved 95.75% accuracy 10 and ANN's accuracy was 91.08%.Furthermore, we also noticed that SDFL obtains a comparable performance with the benchmark method proposed by the authors of UCI database. 7It is worth nothing that the benchmark method is using multiclass support vector machine for classification; whereas SDFL uses a simpler classifier, i.e.Nearest Neighbour (NN) classifier.
Last but not least, it was discerned that the performance of SDFL is on a par with the Cascade Ensemble Learning model (CELearning). 16Both approaches are ensemble learning methods with multiple layers for data learning.The key difference between these approaches is the analysis algorithms in each layer.CELearning is comprised of four  Randomized Trees and the final classification result is obtained through the last layer via the score-level fusion of the four complex classifiers.On the other hand, in SDFL, merely Rayleigh coefficient optimization is implemented to extract the discriminant deep features.Further, a simple classifier, i.e.NN classifier, is adopted in SDFL.This deduces that the discrimination capability of SDFL primarily depends on the SDFL modular model to extract discriminant features demanding no complex classifier.From Table 4, we can notice that SDFL just needs ~4:3 Â 10 À4 seconds per sample (sps) for the training phase and ~4:2 Â 10 À4 sps for the testing phase, on average.The fast feature learning of SDFL and the dimensionality reduction in SDFL to project the data onto a lower-dimensional subspace are the main reasons for having such an efficient computation.

Conclusions
A cascading learning network for human activity recognition using smartphones is proposed.In this network, a chain of independent discriminant learning modules is aggregated, layer by layer in a stackable framework.Each layer is constituted by a discriminant analysis function and a nonlinear activation function to effectively extract the rich features from the inertial data.SDFL network possesses characteristics of good performance even on small-scale training sample sets, as well as less hyper-parameter fine-tuning, and fast computation compared with the other deep learning networks.
Despite showing computational efficiency, the proposed network also demonstrated its classification superiority to most of the state-of-the-art approaches with an accuracy score of ~97% in differentiating human activity classes.In future work, study on small-scale database on SDFL will be investigated.
The motivation and contributions may be rewritten to make it simpler to understand the purpose and achievements of the work (purpose of deep features have to be mentioned). 1.
Any chances of an Intra personal scatter matrix with any other inter personal scattermatrix?How was the threshold or borderline decided?There needs to be explanations with sample values.

2.
The used hardware and software details, and details about dataset should be shifted from result to any other section in methodology.

3.
Slight corrections on technical writings need to be carried out.(eg: Table 4 tabulates: can be 4.

changed as either: Table4 presents or it is tabulated in table 4)
A sample subject wise (Personnel) data table would be nice if presented to know the inter personnel differences.

Are sufficient details of methods and analysis provided to allow replication by others? Partly
If applicable, is the statistical analysis and its interpretation appropriate?I cannot comment.A qualified statistician is required.

Are the conclusions drawn adequately supported by the results? Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: AI, Patterns, Bionics, ML and DEEP Learning, signals,sensors I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Author Response 15 Feb 2022
Ying Han Pang, Multimedia University, Ayer Keroh, Malaysia First of all, we would like to express our heartiest thanks to the Editor-in-Chief and Reviewers who have provided us with many insightful comments and a chance to improve our works.
The motivation and contributions may be rewritten to make it simpler to understand the purpose and achievements of the work (purpose of deep features have to be mentioned).
Response: The motivation and contribution have been revised for better clarification.

1.
Any chances of an Intra personal scatter matrix with any other inter personal scattermatrix?How was the threshold or borderline decided?There needs to be explanations with sample values.

2.
Response: The Method section has been revised for better clarification.The optimal projection in SDFL is computed by optimizing the Rayleigh coefficient for modelling the difference between the classes of data through maximizing the ratio of the interpersonal scatter matrix and intra-personal scatter matrix.The variability between data is contained in a subspace spanned by the eigenvectors corresponding to the number of class -1 largest eigenvalues.Hence, the threshold is C-1 in this work.
The used hardware and software details, and details about dataset should be shifted from result to any other section in methodology.
Response: The amendment has been done and the details have been shifted to the Method section.

3.
Slight corrections on technical writings need to be carried out.(eg: Table 4 tabulates: can be changed as either: Table4 presents or it is tabulated in table 4) Response: The correction has been done.

4.
A sample subject wise (Personnel) data table would be nice if presented to know the inter personnel differences.
Response: Figure 3 is added to illustrate the inter-personnel inertial signal differences of standing activity 5.

Competing Interests:
No competing interests were disclosed.
Reviewer Report 25 November 2021 https://doi.org/10.5256/f1000research.76806.r97058 My concerns are as follows: Objective 2: The authors claim that SDFL learns low-level to higher-level feature representation.This hypothesis should be demonstrated, if possible.Otherwise, I think this objective should be revised accordingly. 1.
Objective 3: The authors should elaborate the subject-independent protocol from the motivation section as a problem statement to be resolved.In addition to that, the advantages of the subject-independent protocol should also be revealed.

2.
The methodology section is ambiguous.
a) For clarity, the data/feature dimensionalities for all mathematical equations should be indicated accordingly.
The experiment section may not be convincing as it is reliant on only a single dataset, and there is no comprehensive analysis conducted.
a) For reproducibility, the parameter configurations should be disclosed.
b) It would be better if an ablation study is presented, e.g., exploration of the number of SDFL layers, etc. (refer to [14 1 ]).Furthermore, the baseline performance, the accuracy for the preprocessed data prior to SDFL learning, is not presented.c) I suggest the authors include additional small-scale HAR datasets for more extensive experiments.
d) I think cross-validation should also be performed.
e) I suggest the authors double-check the results reported for precision, recall, f-score, and accuracy.It is rare that these varying matrices give the same value, approximately 96.3%.Alternatively, the confusion matrices for these evaluation matrices should be disclosed.
(f) In Table 4, I think the inference time should be computed for the SDFL and other conventional models, in place of the pre-processed data?Otherwise, this comparison may not be meaningful.

4.
Partly First of all, we would like to express our heartiest thanks to the Editor-in-Chief and Reviewers who have provided us with many insightful comments and a chance to improve our works.Objective 2: The authors claim that SDFL learns low-level to higher-level feature representation.This hypothesis should be demonstrated, if possible.Otherwise, I think this objective should be revised accordingly.
Response: Thanks for the feedback.Authors have revised Objective 2 for better clarification. 1.
Objective 3: The authors should elaborate the subject-independent protocol from the motivation section as a problem statement to be resolved.In addition to that, the advantages of the subject-independent protocol should also be revealed.
Response: Authors have included the preference of subject-independent solution in real-time applications in the Motivation section for better relating the motivation and the Objective 3.

2.
a) For clarity, the data/feature dimensionalities for all mathematical equations should 3. be indicated accordingly.b) The SDFL hyper-parameters are unknown?
Response: Authors have included dimensionalities of data/feature, number of training classes C and the generated final feature vector in the Methods section for better clarification.Parameters of SDFL are the number of stacking layers, nonlinear activation function, dimensions of intermediate feature vectors and dimensions of the final feature vector.The Method section has been revised to include the information of these parameters for better clarification.a) For reproducibility, the parameter configurations should be disclosed.b) It would be better if an ablation study is presented, e.g., exploration of the number of SDFL layers, etc. (refer to [14 1 ]).Furthermore, the baseline performance, the accuracy for the preprocessed data prior to SDFL learning, is not presented.c) I suggest the authors include additional small-scale HAR datasets for more extensive experiments.d) I think cross-validation should also be performed.e) I suggest the authors doublecheck the results reported for precision, recall, f-score, and accuracy.It is rare that these varying matrices give the same value, approximately 96.3%.Alternatively, the confusion matrices for these evaluation matrices should be disclosed.(f) In Table 4, I think the inference time should be computed for the SDFL and other conventional models, in place of the pre-processed data?Otherwise, this comparison may not be meaningful.(g) In Table 3, the authors compare SDFL with [24], [25], [4], [3], etc., however, not all are reviewed in the related work section.
Response: The information of these parameters has been included in the Methods section for better clarification.Table 3 has been revised to include the benchmark method, that is Multiclass Support Vector Machine, proposed by the original author of the UCI database.This baseline performance which is the accuracy of the preprocessed data with Support Vector Machine has been included in the table.In order to have a better performance comparison with the existing methods, the testing protocol of this study follows the train-test split protocol defined by the database provider, i.e. this database has a version that is already split into training and test sets that contain data from different participants.This paper is using this version of the database.The results have been double-checked and the confusion matrix of SDFL has been included.Table 4 has been revised by just presenting the computational time of SDFL.With this, readers can have a picture of the training and inference time of the proposed SDFL.The Related Work section has been revised to include the reference in Table 3.

4.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com

Figure 1 .
Figure 1.Overview of the proposed Stacked Discriminant Feature Learning (SDFL) system.

Figure 3 .
Figure 3. Different inertial data patterns of standing between two subjects.

Table 1 .
Pre-processed features as input data into the Stacked Discriminant Feature Learning (SDFL) system.

Table 3 .
Performance comparison of Stacked Discriminant Feature Learning (SDFL) with alternative approaches.

the study design appropriate and is the work technically sound? Partly Are sufficient details of methods and analysis provided to allow replication by others? Partly If applicable, is the statistical analysis and its interpretation appropriate? Partly Are all the source data underlying the results available to ensure full reproducibility? Partly Are the conclusions drawn adequately supported by the results? Partly
Competing Interests: No competing interests were disclosed.Reviewer Expertise: Stacked neural networks, deep neural networksI confirm that I