Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system

Sharifah Noor Masidayu Sayed Ismail; Nor Azlina Ab. Aziz; Siti Zainab Ibrahim; Sophan Wahyudi Nawawi; Salem Alelyani; Mohamed Mohana; Lee Chia Chun

doi:10.12688/f1000research.73255.1

Home Browse Evaluation of electrocardiogram: numerical vs. image data for emotion...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system

[version 1; peer review: 2 approved with reservations, 1 not approved]

Sharifah Noor Masidayu Sayed Ismail ¹, Nor Azlina Ab. Aziz ², Siti Zainab Ibrahim¹, [...] Sophan Wahyudi Nawawi³, Salem Alelyani^4,5, Mohamed Mohana⁴, Lee Chia Chun⁶

Sharifah Noor Masidayu Sayed Ismail ¹, Nor Azlina Ab. Aziz ², [...] Siti Zainab Ibrahim¹, Sophan Wahyudi Nawawi³, Salem Alelyani^4,5, Mohamed Mohana⁴, Lee Chia Chun⁶

PUBLISHED 04 Nov 2021

Author details Author details

¹ Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, Melaka, 75450, Malaysia
² Faculty of Engineering, Multimedia University, Bukit Beruang, Melaka, 75450, Malaysia
³ School of Electrical Engineering, Faculty of Engineering, Universiti Teknologi Malaysia, Skudai, Johor Bahru, 81310, Malaysia
⁴ Center for Artificial Intelligence, King Khalid University, Abha, 61421, Saudi Arabia
⁵ College of Computer Science, King Khalid University, Abha, 61421, Saudi Arabia
⁶ Hexon Data Sdn Bhd, Kuala Lumpur, 59200, Malaysia

Sharifah Noor Masidayu Sayed Ismail
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Software, Visualization, Writing – Original Draft Preparation

Nor Azlina Ab. Aziz
Roles: Conceptualization, Data Curation, Funding Acquisition, Investigation, Project Administration, Resources, Software, Supervision, Validation, Writing – Review & Editing

Siti Zainab Ibrahim
Roles: Conceptualization, Data Curation, Investigation, Project Administration, Resources, Software, Supervision, Validation, Writing – Review & Editing

Sophan Wahyudi Nawawi
Roles: Funding Acquisition, Supervision, Validation

Salem Alelyani
Roles: Funding Acquisition, Supervision, Validation

Mohamed Mohana
Roles: Funding Acquisition, Supervision, Validation

Lee Chia Chun
Roles: Funding Acquisition, Supervision, Validation

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research Synergy Foundation gateway.

Abstract

Background: The electrocardiogram (ECG) is a physiological signal used to diagnose and monitor cardiovascular disease, usually using ECG wave images. Numerous studies have proven that ECG can be used to detect human emotions using numerical data; however, ECG is typically captured as a wave image rather than as a numerical data. There is still no consensus on the effect of the ECG input format (either as an image or a numerical value) on the accuracy of the emotion recognition system (ERS). The ERS using ECG images is still inadequately studied. Therefore, this study compared ERS performance using ECG image and ECG numerical data to determine the effect of the ECG input format on the ERS.
Methods: This study employed the DREAMER dataset, which contains 23 ECG recordings obtained during audio-visual emotional elicitation. Numerical data was converted to ECG images for the comparison. Numerous approaches were used to obtain ECG features. The Augsburg BioSignal Toolbox (AUBT) and the Toolbox for Emotional feature extraction from Physiological signals (TEAP) extracted features from numerical data. Meanwhile, features were extracted from image data using Oriented FAST and rotated BRIEF (ORB), Scale Invariant Feature Transform (SIFT), KAZE, Accelerated-KAZE (AKAZE), Binary Robust Invariant Scalable Keypoints (BRISK), and Histogram of Oriented Gradients (HOG). Dimension reduction was accomplished using linear discriminant analysis (LDA), and valence and arousal were classified using the Support Vector Machine (SVM).
Results: The experimental results indicated that numerical data achieved arousal and valence accuracy of 69% and 79%, respectively, which was greater than those of image data. For ECG images, the highest accuracy for arousal was 58% percent; meanwhile, the valence was 63%.
Conclusions: The finding showed that numerical data provided better accuracy for ERS. However, ECG image data which shows positive potential and can be considered as an input modality for the ERS.

Keywords

Emotion recognition, electrocardiogram, numerical ECG, image ECG, DREAMER

Corresponding authors: Sharifah Noor Masidayu Sayed Ismail, Nor Azlina Ab. Aziz, Siti Zainab Ibrahim

Competing interests: No competing interests were disclosed.

Grant information: This project is funded by the TM Research & Development Grant (RDTC/190988), awarded to Multimedia University (MMU).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2021 Sayed Ismail SNM et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Sayed Ismail SNM, Ab. Aziz NA, Ibrahim SZ et al. Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system [version 1; peer review: 2 approved with reservations, 1 not approved]. F1000Research 2021, 10:1114 (https://doi.org/10.12688/f1000research.73255.1) First published: 04 Nov 2021, 10:1114 (https://doi.org/10.12688/f1000research.73255.1) Latest published: 30 May 2022, 10:1114 (https://doi.org/10.12688/f1000research.73255.2)

Introduction

Medical professionals have been actively using electrocardiogram (ECG) wave images as a tool for monitoring¹^,² and diagnosing³^–⁶ cardiovascular diseases, such as heart attacks, dysrhythmia, and pericarditis, with some reported accuracy of more than 99% in the past decade. Besides monitoring and diagnosing health-related diseases, many studies have proven that human emotions can also be identified using ECG in the form of numerical data.⁷^–¹⁰

The effects of using different types of ECG inputs to recognise emotions by the emotion recognition system (ERS) have yet to be closely studied. In addition, there is no consensus on whether or not the type of ECG input format affects the emotion classification accuracy by the ERS. Most researchers have focused on recognising emotions using ECG numerical data instead of using EGC wave images. To date, research on the use of ECG wave images in identifying emotions is still absent. Therefore, to address this gap, the objective of this study is to compare emotion classification performance using ECG image and ECG numerical data to determine the effect of the ECG input format on the ERS.

Emotion model

Emotions can be seen in two different models put forward by Paul Ekman¹¹ and James Russell,¹² namely, the discrete emotion and dimensional emotion models. Ekman, a psychologist, suggested six basic emotions: happiness, sadness, anger, fear, disgust, and surprise. On the other hand, Russell presented a two-dimensional scale of emotions consisting of valence and arousal (Figure 1). Valence refers to positive or negative feelings, and arousal indicates the intensity of the feeling, either high or low. This study used the latter emotion model to classify the subject’s emotions, because 1) the work presented in¹³ used the same emotion model; hence, allowing benchmarking of the performance, and 2) simplicity of binary classification.

Figure 1. The valence arousal scale.¹⁴

Electrocardiogram & Emotion

An electrocardiogram is used to measure electrical activity in the human heart by attaching electrodes to the human body. The standard is a 12-lead ECG system. However, today ECG devices have evolved from bulky nonportable devices to wearable portable devices. The accuracy of the signal by portable devices is comparable to conventional medical devices. This suggests that researchers can use wearable ECG devices for purposes similar to conventional devices, including for studying human emotions. However, most of these devices store the ECG as images instead of raw numerical data.

The ECG signals have P, Q, R, S, and T waves (Figure 2). Emotional states are associated with autonomic nervous system's (ANS) physiological responses.¹⁵ Different emotions influence human heart activities differently; these influences may be hidden in the ECG wave.¹⁶ These responses can be detected through ECG by monitoring the main features of ECG, namely, heart rate (HR) and heart rate variability (HRV).

Figure 2. The P wave, QRS complex, and T wave in the standard electrocardiogram (ECG).¹⁷

This study used ECG numerical data obtained from the multimodal database called the DREAMER dataset.¹³ In the dataset, electroencephalogram (EEG) and ECG signals were recorded from 23 participants during an emotion elicitation session. The emotions were elicited by using 18 audio-visual stimulations. Valence and arousal emotion model was used to classify the elicited emotions. This paper only focuses on the use of ECG. We employed the Augsburg BioSignal Toolbox (AUBT)¹⁸ and the Toolbox for Emotional feature extraction from Physiological signals (TEAP)¹⁹ to facilitate feature extraction from the ECG numerical data. Then, linear discriminant analysis (LDA) was applied to reduce the dimension of the extracted ECG numerical features.

Since the DREAMER dataset only has numerical data, the data must be converted into the corresponding ECG wave images for comparison purposes. Six different feature extractors, namely, Oriented FAST and rotated BRIEF (ORB), Scale Invariant Feature Transform (SIFT), KAZE, Accelerated-KAZE (AKAZE), Binary Robust Invariant Scalable Keypoints (BRISK), and Histogram of Oriented Gradients (HOG), were applied to the ECG wave images to detect and extract features. The Support Vector Machine (SVM) was used to classify the valence and arousal of both ECG numerical features and ECG image features.

In the following section, an overview of related works on the ERS is presented. We then describe the selected dataset and explain the proposed methods in detail. This is followed by the results and a discussion and conclusions section.

Related works

Researchers in the emotion recognition field have been proposing multiple approaches using electrocardiogram signals. For instance, Minhad, Ali, and Reaz²⁰ used ECG numerical data to classify emotions of happiness and anger. They achieved 83.33% accuracy using the SVM classification method. Besides, Tivatansakul and Ohkura²¹ used ECG numerical data from the AUBT dataset to detect emotions for the emotional healthcare system. K-Nearest Neighbour (KNN) successfully classified three emotions (joy, anger, and sadness) with an accuracy 85.75%, 82.75%, and 95.25%, respectively.

Katsigiannis and Ramzan suggested that ERS should use low-cost and off-the-shelf devices to collect ECG signals based on numerical format.¹³ AUBT and Biosig Toolbox were used to extract the signal features. Classification using SVM with a radial basis function kernel successfully achieved 62.37% for valence and arousal. The MPED database for ERS was proposed by Song et al.²² using ECG numerical data to recognise discrete emotions (joy, humour, disgust, anger, fear, sadness, and neutrality). Attention Long Short-Term Memory (A-LSTM) was used as a feature extractor to extract the frequency and time-domain features from the physiological signal. The A-LSTM was used as a classifier along with SVM, KNN, and Long Short-Term Memory (LSTM). Averagely, A-LSTM achieved better results of 40% to 55% compared to those of other classifiers.

Just as with the widespread use of numerical ECG in human emotion studies, ECG images are also widely used to identify cardiovascular-related diseases. For example, Hao et al.²³ used ECG images to detect and classify myocardial infarction (MI). MI is a disease caused by severe cardiovascular obstruction that leads to irreversible injury or even death. KNN and SVM were used in this study and achieved 89.84% and 92.19%, respectively. Besides, Mandal, Mondal, and Roy²⁴ used ECG images to detect ventricular arrhythmia (VA), such as ventricular tachycardia (VT) and ventricular fibrillation (VF) in infants and children. SVM, KNN, and random forest (RF) were used in this study and successfully achieved 93.11%, 92.56%, and 95.36%, respectively.

Although much research has been conducted using ECG for ERS, most of them focused mainly on numerical data analysis instead of ECG wave images. However, systems based on ECG images have achieved excellent results in detecting cardiovascular-related diseases. As mentioned before, it remains uncertain whether the ECG input format, numerical value or wave image, affects the emotional classification accuracy in the ERS. Therefore, it is essential to explore the ERS using different input formats of ECG to address this knowledge gap.

Methods

Proposed method

The proposed method consists of three stages: feature extraction, feature dimension reduction, and emotion classification. The data of the present study were obtained in the experiment described in the original study.¹³ The current study began in September 2020. Matlab version 9.7 was utilized for data conversion and feature extraction, whereas Python version 3.8.5 was used for feature dimension reduction (numerical) and classification. The overall structure of the proposed method is illustrated in Figure 3. The analysis code used in this study is available from GitHub and archived with Zenodo.⁴¹

Figure 3. Structure of emotion recognition system: (a) numerical data and (b) image data.

The dataset (DREAMER)

We built our ERS on a publicly accessible database consisting of ECG signals recorded from 23 participants during emotion elicitation by audio-visual stimuli. The ECG was recorded using the SHIMMER ECG sensor at 256 Hz. Nine emotions: calmness, surprise, amusement, fear, excitement, disgust, happiness, anger, and sadness were elicited using 18 video excerpts.

The total data amount is 414 data (23 subjects x 18 videos). As previously mentioned, this work is only interested in ECG signals; hence, EEG signals are not included in this study. Additionally, we did not use the dominance rating score, since Russell’s two-dimensional emotional model is adapted here to classify emotions. The summary of the DREAMER dataset is tabulated in Table 1.

Table 1. The summary of the dataset.

No of subject	23
No of videos	18 audio-visual stimuli
Type of stimuli	Audio-video
Used Signal (Hz)	ECG (256)
Rating scales	Valence, Arousal
Rating values	1 – 5

Experimental setup

1) ECG wave image

The ECG numerical data was converted into ECG wave images preceding the analysis of ECG wave images using MATLAB version 9.7 (Figure 4). Using Python version 3.8.5, the converted ECG images were then resized to 60% of the original size to reduce the computational time. After resizing, the coloured images were converted into greyscale images. Then, binarization of the image using a threshold was done. Automatic image thresholding, Otsu’s method,²⁵ was used here. Otsu’s method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance. This method provides the best performance, as stated in.²⁶

Figure 4. The electrocardiogram (ECG) wave image converted from ECG numerical data.

In total, six different feature extractors were applied to extract features from processed ECG images. The feature extractors are as follows: ORB,²⁷ SIFT,²⁸ KAZE,²⁹ AKAZE,³⁰ BRISK,³¹ and HOG.³² All of them successfully extracted the ECG features, including the peaks, edges, and corners. However, some feature extractors, such as ORB and SIFT, failed to detect important features, particularly the R-peaks, due to the presence of mass noise,³³ which is believed to have affected emotional classification (Figure 5). The extracted images were then given to the classifier (SVM) to classify valence and arousal. This whole process of feature extraction and classification was done using Python version 3.8.5.

Figure 5. Oriented FAST and rotated BRIEF (ORB) is unable to detect the electrocardiogram (ECG) peaks due to the presence of noise.

ECG numerical data

Numerical ECG data did not go through any pre-processing process as suggested in¹³ due to being less susceptible to interference owing to their higher voltage amplitudes. Two open-source toolboxes, namely, Augsburg BioSignal Toolbox (AUBT)¹⁸ and Toolbox for Emotional feature extraction from Physiological signals (TEAP),¹⁹ were employed to facilitate feature extraction from the ECG signals. Both of them successfully extracted 81 features (Table 2) and 16 features (Table 3) from the ECG signals. The extracted features included heart rate variability (HRV), inter-beat interval (IBI), tachogram power, and statistical features such as mean, median, and standard deviation. The dimension of the features was reduced using linear discriminant analysis, one of the well-known feature reduction methods.³⁴ This process was performed to reduce the computational cost and to improve the separation of emotion classes.³⁵ The extracted features were then given to the classifier (SVM) to classify valence and arousal.

Table 2. Features extracted from Augsburg Bio-signal Toolbox (AUBT).

Features	Description
P, Q, R, S, T	P-, Q-, R-, S-, T-peaks (ECG)
HRV	Heart rate variability
Ampl	Amplitude Signal
Mean	Mean value
Median	Median value
Std	Standard deviation
Min	Minimum value
Max	Maximum value
SpecRange	Mean of the frequency spectrum in a given range

Table 3. Features extracted from Toolbox for Emotional feature extraction from Physiological signals (TEAP).

Features	Description
meanIBI	Mean inter-beat interval
HRV	Heart Rate Variability
MSE	Multiscale entropy at 5 levels
sp0001/0102/0203/0304	Spectral power 0-0.1Hz, 0.1-0.2Hz, 0.2-0.3Hz, 0.3-0.4Hz
energyRatio	Spectral energy ratio between f<0.08Hz/f>0.15Hz and f<5.0Hz
tachogram_LF/MF/HF	Spectral power in tachogram (HRV) for low, medium, and high frequencies.
tachogram_energy_ratio	Energy ratio for tachogram spectral content (MF/(LF+HF))

Support vector machine

Classification was performed using SVM. The SVM works by separating the class data points and drawing a boundary called the hyperplane between them. Additionally, SVM has a low computational cost and shows excellent performance in classifying emotions, as reported in previous studies.¹³^,³⁶^,³⁷ The data was then divided into training and test sets with a ratio of 80:20. The parameters for SVM were tuned using GridSearchCV.³⁸ As we had a small data size, we used 10-fold cross-validation to improve ERS performance. The emotions were determined as follows: high/low valence and high/low arousal, based on the participants’ self-assessment rating.

Results

The experimental results for numerical data showed that the accuracy of arousal achieved using the TEAP feature extractor (69%) was higher than that of the AUBT feature (64%). However, the TEAP feature managed to obtain 67% valence accuracy, while the AUBT feature recorded up to 79%. These results are better than what were recorded in Ref. 11.

The classification results using ECG wave images recorded an arousal accuracy of 53% to 58%. The highest result was achieved by the SIFT feature, followed by ORB, HOG, KAZE, BRISK, and the AKAZE features. Meanwhile, the highest accuracy for valence was attained by the KAZE feature with 63%, followed by HOG, BRISK, AKAZE, SIFT, and lastly, ORB with 48%, the lowest among other features. The results of this study are presented in Table 4.

Table 4. Testing emotion classification accuracy for electrocardiogram (ECG) numerical data and ECG image.

Type of ECG	Feature extractor	Arousal accuracy	Valence accuracy
Image	ORB	0.57	0.48
	SIFT	0.58	0.51
	AKAZE	0.53	0.54
	BRISK	0.54	0.58
	HOG	0.57	0.60
	KAZE	0.54	0.63
Numerical data	TEAP	0.69	0.67
	AUBT	0.64	0.79
	AUBT*	0.62	0.62

* Accuracy of DREAMER reported in the original paper.

Discussion & conclusions

Findings showed that numerical data provided better accuracy for ERS compared to ECG images. In addition, numerical data was easier to handle and process compared to image data. Moreover, the results obtained here using ECG numerical data were even better than those reported by DREAMER.¹³ This is contributed by the additional processes in our proposed method, the feature reduction using LDA, which was not included in the DREAMER paper. LDA plays an essential role in improving the performance of the emotion recognition system.³⁵^,³⁹ On top of that, it is worth noting that the results obtained using ECG image data also showed positive potential and could be considered as an input modality for the ERS. The features extracted by KAZE provided 63% accuracy for valence, which is better than the original work in Ref. 11. Hence, ECG images are recommended for building ERS. ECG images are attractive as the format allows usage of many image-based methods such as image augmentation to increase the data size, the convolution neural networks (CNN), and application of transfer learning from models trained using large data.

However, some limitations were found throughout the study and needed to be addressed to achieve better emotion classification results. The first limitation is, as per suggestion by the DREAMER paper, we did not run the pre-process process to the signal leading to the presence of noise in the signal, both ECG data format, which have affected emotion classification, especially for image data. Therefore, the use of filtering and noise reduction in the pre-processing stage should be considered. The second limitation is the data size, which is too small for image learning and classification, leading to lower accuracy.⁴⁰ In the future, with a larger data size, researchers can consider deep learning techniques for emotion classification using ECG images as a primary modality.

To conclude, ECG numerical data provided a better performance of emotion classification. In addition, ECG image data that shows positive potential, thus it can be considered an input modality for the ERS in future studies.

Data availability

Source data

The DREAMER dataset was first presented here: https://doi.org/10.1109/JBHI.2017.2688239 and can be found on Zenodo. Access is restricted and users are required to apply. The decision whether to grant/deny access is solely under the responsibility of the record owner.

Extended data

Analysis code available from: https://github.com/nr-isml/ECG-Numerical-Vs.-Image-Data-for-Emotion-Recognition-System

Archived analysis code as at time of publication: https://doi.org/10.5281/zenodo.5542739.⁴¹

License: Data are available under the terms of the Creative Commons Zero “No rights reserved” data waiver (CC0 1.0 Public domain dedication).

Acknowledgements

The authors would like to thank those who were involved in this experiment, either directly or indirectly.

References

1. Mena LJ, et al.: Mobile Personal Health Monitoring for Automated Classification of Electrocardiogram Signals in Elderly. Comput. Math. Methods Med. 2018; 2018(Figure 1): 1–9.
2. Wang YH, Chung CG, Lin CC, et al.: The Study of the Electrocardiography Monitoring for the Elderly Based on Smart Clothes. 8th Int. Conf. Inf. Sci. Technol. 2018; pp. 478–482.
3. Ullah A, Anwar SM, Bilal M, et al.: Classification of arrhythmia by using deep learning with 2-D ECG spectral image representation. Remote Sens. 2020; 12(10): 1–14. Publisher Full Text .
4. Tayel MB, El-Bouridy ME: ECG images classification using artificial neural network based on several feature extraction methods. 2008 Int. Conf. Comput. Eng. Syst. ICCES 2008. 2008; pp. 113–115.
5. Mohamed B, Issam A, Mohamed A, et al.: ECG Image Classification in Real time based on the Haar-like Features and Artificial Neural Networks. Procedia Comput. Sci. 2015; 73(Awict): 32–39. Publisher Full Text
6. Wang F, Syeda-Mahmood T, Beymer D: Finding disease similarity by combining ECG with heart auscultation sound. Comput. Cardiol. 2007; 34: 261–264.
7. Soleymani M, Lichtenauer J, Pun T, et al.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 2012; 3(1): 42–55. Publisher Full Text
8. Abadi MK, Subramanian R, Kia SM, et al.: DECAF: MEG-Based Multimodal Database for Decoding Affective Physiological Responses. IEEE Trans. Affect. Comput. 2015; 6(3): 209–222. Publisher Full Text
9. Subramanian R, Wache J, Abadi MK, et al.: ASCERTAIN: Emotion and personality recognition using commercial sensors. IEEE Trans. Affect. Comput. 2018; 9(2): 147–160. Publisher Full Text
10. Siddharth S, Jung T-P, Sejnowski TJ: Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing. IEEE Trans. Affect. Comput. 2019; 1–1. Publisher Full Text
11. Ekman P: Cognition & Emotion An argument for basic emotions An Argument for Basic Emotions. COGNlTfON Emot. 1992; 6(May 2013): 169–200. Publisher Full Text
12. Russell JA: A circumplex model of affect. J. Pers. Soc. Psychol. 1980; 39(6): 1161–1178. Publisher Full Text
13. Katsigiannis S, Ramzan N: DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals from Wireless Low-cost Off-the-Shelf Devices. IEEE J. Biomed. Heal. Informatics. . 2018; 22(1): 98–107. PubMed Abstract | Publisher Full Text
14. Yu LC, et al.: Building Chinese affective resources in valence-arousal dimensions. 2016 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. NAACL HLT 2016 - Proc. Conf. 2016; (no. June): pp. 540–545.
15. Bota PJ, Wang C, Fred ALN, et al.: A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals. IEEE Access. 2019; 7: 140990–141020. Publisher Full Text
16. Wen WH, Qiu YH, Liu GY: Electrocardiography recording, feature extraction and classification for emotion recognition. 2009 WRI World Congr. Comput. Sci. Inf. Eng. CSIE 2009. 2009; vol. 4: pp. 168–172.
17. C. U. First Faculty of Medicine: Electrocardiogram - WikiLectures.2018. [Accessed: 04-Oct-2021]. Reference Source
18. Wagner J: Augsburg biosignal toolbox (aubt). Univ. Augsbg; 2014.
19. Soleymani M, Villaro-Dixon F, Pun T, et al.: Toolbox for Emotional feAture extraction from Physiological signals (TEAP). Front. ICT. 2017; 4(FEB): 1–7. Publisher Full Text
20. Minhad KN, Ali SHM, Reaz MBI: Happy-anger emotions classifications from electrocardiogram signal for automobile driving safety and awareness. J. Transp. Heal. 2017; 7(November): 75–89.
21. Tivatansakul S, Ohkura M: Emotion Recognition using ECG Signals with Local Pattern Description Methods. Int. J. Affect. Eng. 2015; 15(2): 51–61. Publisher Full Text
22. Song T, Zheng W, Lu C, et al.: MPED: A multi-modal physiological emotion database for discrete emotion recognition. IEEE Access. 2019; 7(October): 12177–12191. Publisher Full Text
23. Hao P, Gao X, Li Z, et al.: Multi-branch fusion network for Myocardial infarction screening from 12-lead ECG images. Comput. Methods Programs Biomed. 2020; 184: 105286.
24. Mandal S, Mondal P, Roy AH: Detection of Ventricular Arrhythmia by using Heart rate variability signal and ECG beat image. Biomed. Signal Process. Control. 2021; 68(May): 102692. Publisher Full Text
25. Otsu N: A Threshold Selection Method from Gray-Level Histograms. IEEE Trans. Syst. Man Cybern. 1979; 9(1): 62–66. Publisher Full Text
26. Trier OD, Taxt T: Evaluation of Binarization Methods for Document Images. IEEE Trans. Pattern Anal. Mach. Intell. 1995; 17(3): 312–315. Publisher Full Text
27. Rublee E, Rabaud V, Konolige K, et al.: ORB: An efficient alternative to SIFT or SURF. Proc. IEEE Int. Conf. Comput. Vis. 2011; (May): 2564–2571.
28. Shi Y, Lv Z, Bi N, et al.: An improved SIFT algorithm for robust emotion recognition under various face poses and illuminations. Neural Comput. Appl. 2020; 32(13): 9267–9281. Publisher Full Text
29. Alcantarilla PF, Bartoli A, Davison AJ: KAZE features. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 2012; vol. 7577 LNCS(no. PART 6): pp. 214–227.
30. Tareen SAK, Saleem Z: A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK. 2018 Int. Conf. Comput. Math. Eng. Technol. Inven. Innov. Integr. Socioecon. Dev. iCoMET 2018 - Proc. 2018; vol. 2018-Janua: pp. 1–10.
31. Liu Y, Zhang H, Guo H, et al.: A FAST-BRISK feature detector with depth information. Sensors (Switzerland). 2018; 18(11). Publisher Full Text
32. Rathikarani V, Dhanalakshmi P, Vijayakumar K: Automatic ECG Image Classification Using HOG and RPC Features by Template Matching.2016; pp. 117–125.
33. Ma Y, Wang Z, Wu C: Feature extraction from noisy image using PCNN. Proc. IEEE ICIA 2006-2006 IEEE Int. Conf. Inf. Acquis. 2006; (no. September): pp. 808–813.
34. Valenzi S, Islam T, Jurica P, et al.: Individual Classification of Emotions Using EEG. J. Biomed. Sci. Eng. 2014; 07(08): 604–620. Publisher Full Text
35. Velliangiri S, Alagumuthukrishnan S, Thankumar Joseph SI: A Review of Dimensionality Reduction Techniques for Efficient Computation. Procedia Comput. Sci. 2019; 165: 104–111. Publisher Full Text
36. Bulagang AF, Weng NG, Mountstephens J, et al.: A review of recent approaches for emotion classification using electrocardiography and electrodermography signals. Informatics Med. Unlocked. 2020; 20: 100363. Publisher Full Text
37. Zhai J, Barreto A: Stress detection in computer users based on digital signal processing of noninvasive physiological variables. Annu. Int. Conf. IEEE Eng. Med. Biol. - Proc. 2006; (no. May): pp. 1355–1358.
38. Pedregosa F, Varoquaux G, Gramfort A, et al.: Scikit-learn: Machine Learning in Python. J. Machine Learn. Res. 2011; 12: 2825–2830.
39. Haq S, Jackson P: Multimodal emotion recognition. In: Machine audition: principles, algorithms and systems. Multimodal Emot. Recognit. 2010.
40. Mitsa T: How Do You Know You Have Enough Training Data?. Towards Data Science. 2019. [Accessed: 07-Jun-2021]. Reference Source
41. nr-isml: nr-isml/ECG-Numerical-Vs.-Image-Data-for-Emotion-Recognition-System: First release (ECG). Zenodo. 2021.Publisher Full Text

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 04 Nov 2021

Author details Author details

Sharifah Noor Masidayu Sayed Ismail
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Software, Visualization, Writing – Original Draft Preparation

Nor Azlina Ab. Aziz
Roles: Conceptualization, Data Curation, Funding Acquisition, Investigation, Project Administration, Resources, Software, Supervision, Validation, Writing – Review & Editing

Siti Zainab Ibrahim
Roles: Conceptualization, Data Curation, Investigation, Project Administration, Resources, Software, Supervision, Validation, Writing – Review & Editing

Sophan Wahyudi Nawawi
Roles: Funding Acquisition, Supervision, Validation

Salem Alelyani
Roles: Funding Acquisition, Supervision, Validation

Mohamed Mohana
Roles: Funding Acquisition, Supervision, Validation

Lee Chia Chun
Roles: Funding Acquisition, Supervision, Validation

Competing interests

No competing interests were disclosed.

Grant information

This project is funded by the TM Research & Development Grant (RDTC/190988), awarded to Multimedia University (MMU).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (2)

version 2

Revised

Published: 30 May 2022, 10:1114

https://doi.org/10.12688/f1000research.73255.2

version 1

Published: 04 Nov 2021, 10:1114

https://doi.org/10.12688/f1000research.73255.1

© 2021 Sayed Ismail SNM et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Sayed Ismail SNM, Ab. Aziz NA, Ibrahim SZ et al. Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system [version 1; peer review: 2 approved with reservations, 1 not approved]. F1000Research 2021, 10:1114 (https://doi.org/10.12688/f1000research.73255.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 1

VERSION 1

PUBLISHED 04 Nov 2021

Views

Reviewer Report 16 Mar 2022

Md. Asadur Rahman, Department of Biomedical Engineering, Military Institute of Science and Technology (MIST), Dhaka, Bangladesh

Approved with Reservations

https://doi.org/10.5256/f1000research.76896.r121059

Figure 2 should be redrawn, only providing the reference is not enough to present a figure or image in the article.
The ECG signals used in this work contain some baseline wandering. It should be removed before further analysis. A simple technique with Matlab code is described in Rahman et al (2019)¹ to remove baseline wander. I am expecting a result comparing the emotion recognition rate before and after the baseline wander removal from the ECG signal.
In addition to that, is it possible to use LSTM to the ECG time series to find the accuracy of the emotion recognition from ECG signals?
Please provide detailed information about transforming the signal to image conversion.
The reference-related comparison and discussion should be completed in the result and discussions. If possible, avoid referencing in the Conclusion section.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Rahman M, Milu M, Anjum A, Siddik A, et al.: A statistical designing approach to MATLAB based functions for the ECG signal preprocessing. Iran Journal of Computer Science. 2019; 2 (3): 167-178 Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Biomedical Signal Processing

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 30 May 2022

Sharifah Noor Masidayu Sayed Ismail, Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, 75450, Malaysia

30 May 2022

Author Response

The paper titled “Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system” looks interesting. Although it is a short paper, I found it interesting and am positive about ... Continue reading The paper titled “Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system” looks interesting. Although it is a short paper, I found it interesting and am positive about this work, but it should go through some modification:

We appreciate your feedback and suggestions. We have revised the manuscript as needed.

Figure 2 should be redrawn, only providing the reference is not enough to present a figure or image in the article.

Thank you for your suggestions. As per your recommendations, we have redrawn the figure and an additional note has been added in the legend. As several modifications were made to the manuscript, Figure 2 was originally changed to Figure 1. This figure can be found on page 4.

The ECG signals used in this work contain some baseline wandering. It should be removed before further analysis. A simple technique with MATLAB code is described in Rahman et al (2019)1 to remove baseline wander. I am expecting a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal.

Thank you for your suggestion. We did use the method suggested as cited accordingly. However, your expectation of having a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal is inappropriate for our work because the focus of this paper is to compare emotion classification performance using 1-D and 2-D ECGs to investigate the effect of the ECG input format on the ERS.

In addition to that, is it possible to use LSTM to the ECG time series to find the accuracy of the emotion recognition from ECG signals?

As far as we know, it is possible to use LSTM on the ECG time series to find the accuracy of emotion recognition from ECG signals. The work by Song et al. used LSTM for the same purpose as your concern. Below is the paper, which you can take a look at. We hope it answers your question.

T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, and Z. Cui, “MPED: A multi-modal physiological emotion database for discrete emotion recognition,” IEEE Access, vol. 7, no. October, pp. 12177–12191, 2019.

Please provide detailed information about transforming the signal to image conversion.

Thank you for pointing this out. We have revised the 2-D ECG subsection, where the pre-processing process until the transformation from 1-D to 2-D is explained in detail. This subsection can be found on pages 11 and 12, which can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

The reference-related comparison and discussion should be completed in the result and discussions. If possible, avoid referencing in the Conclusion section.

Thank you for bringing this to our attention. We found your comments extremely helpful and have revised them accordingly. The revised text can be read as follows:

“The results indicate that both inputs work comparably well in classifying emotions. This finding is demonstrated by the fact that the best valence performance was obtained using a 1-D ECG, and the best arousal performance was acquired using a 2-D ECG. Additionally, ERS with 1-D ECG was combined with dimensionality reduction, called LDA. The presence of LDA improved the ERS performance in valence emotion but not in arousal. In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.

However, it is worth mentioning that the results obtained using 2-D ECG demonstrated potential for use as an input modality for the ERS. Additionally, 2-D ECGs are appealing because the format enables the use of a variety of image-based methods such as image augmentation to increase the data size, convolution neural networks (CNN), and the application of transfer learning from models trained using large data. To summarise, the ERS performance of the two ECG inputs is comparable since both yield a promising outcome for emotion recognition.”
The paper titled “Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system” looks interesting. Although it is a short paper, I found it interesting and am positive about this work, but it should go through some modification:

We appreciate your feedback and suggestions. We have revised the manuscript as needed.

Figure 2 should be redrawn, only providing the reference is not enough to present a figure or image in the article.

Thank you for your suggestions. As per your recommendations, we have redrawn the figure and an additional note has been added in the legend. As several modifications were made to the manuscript, Figure 2 was originally changed to Figure 1. This figure can be found on page 4.

The ECG signals used in this work contain some baseline wandering. It should be removed before further analysis. A simple technique with MATLAB code is described in Rahman et al (2019)1 to remove baseline wander. I am expecting a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal.

Thank you for your suggestion. We did use the method suggested as cited accordingly. However, your expectation of having a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal is inappropriate for our work because the focus of this paper is to compare emotion classification performance using 1-D and 2-D ECGs to investigate the effect of the ECG input format on the ERS.

In addition to that, is it possible to use LSTM to the ECG time series to find the accuracy of the emotion recognition from ECG signals?

As far as we know, it is possible to use LSTM on the ECG time series to find the accuracy of emotion recognition from ECG signals. The work by Song et al. used LSTM for the same purpose as your concern. Below is the paper, which you can take a look at. We hope it answers your question.

T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, and Z. Cui, “MPED: A multi-modal physiological emotion database for discrete emotion recognition,” IEEE Access, vol. 7, no. October, pp. 12177–12191, 2019.

Please provide detailed information about transforming the signal to image conversion.

Thank you for pointing this out. We have revised the 2-D ECG subsection, where the pre-processing process until the transformation from 1-D to 2-D is explained in detail. This subsection can be found on pages 11 and 12, which can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

The reference-related comparison and discussion should be completed in the result and discussions. If possible, avoid referencing in the Conclusion section.

Thank you for bringing this to our attention. We found your comments extremely helpful and have revised them accordingly. The revised text can be read as follows:

“The results indicate that both inputs work comparably well in classifying emotions. This finding is demonstrated by the fact that the best valence performance was obtained using a 1-D ECG, and the best arousal performance was acquired using a 2-D ECG. Additionally, ERS with 1-D ECG was combined with dimensionality reduction, called LDA. The presence of LDA improved the ERS performance in valence emotion but not in arousal. In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.

However, it is worth mentioning that the results obtained using 2-D ECG demonstrated potential for use as an input modality for the ERS. Additionally, 2-D ECGs are appealing because the format enables the use of a variety of image-based methods such as image augmentation to increase the data size, convolution neural networks (CNN), and the application of transfer learning from models trained using large data. To summarise, the ERS performance of the two ECG inputs is comparable since both yield a promising outcome for emotion recognition.”
Competing Interests: The authors declare that they have no conflict of interest. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 30 May 2022

Sharifah Noor Masidayu Sayed Ismail, Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, 75450, Malaysia

30 May 2022

Author Response

The paper titled “Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system” looks interesting. Although it is a short paper, I found it interesting and am positive about ... Continue reading The paper titled “Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system” looks interesting. Although it is a short paper, I found it interesting and am positive about this work, but it should go through some modification:

We appreciate your feedback and suggestions. We have revised the manuscript as needed.

Figure 2 should be redrawn, only providing the reference is not enough to present a figure or image in the article.

Thank you for your suggestions. As per your recommendations, we have redrawn the figure and an additional note has been added in the legend. As several modifications were made to the manuscript, Figure 2 was originally changed to Figure 1. This figure can be found on page 4.

The ECG signals used in this work contain some baseline wandering. It should be removed before further analysis. A simple technique with MATLAB code is described in Rahman et al (2019)1 to remove baseline wander. I am expecting a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal.

Thank you for your suggestion. We did use the method suggested as cited accordingly. However, your expectation of having a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal is inappropriate for our work because the focus of this paper is to compare emotion classification performance using 1-D and 2-D ECGs to investigate the effect of the ECG input format on the ERS.

In addition to that, is it possible to use LSTM to the ECG time series to find the accuracy of the emotion recognition from ECG signals?

As far as we know, it is possible to use LSTM on the ECG time series to find the accuracy of emotion recognition from ECG signals. The work by Song et al. used LSTM for the same purpose as your concern. Below is the paper, which you can take a look at. We hope it answers your question.

T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, and Z. Cui, “MPED: A multi-modal physiological emotion database for discrete emotion recognition,” IEEE Access, vol. 7, no. October, pp. 12177–12191, 2019.

Please provide detailed information about transforming the signal to image conversion.

Thank you for pointing this out. We have revised the 2-D ECG subsection, where the pre-processing process until the transformation from 1-D to 2-D is explained in detail. This subsection can be found on pages 11 and 12, which can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

The reference-related comparison and discussion should be completed in the result and discussions. If possible, avoid referencing in the Conclusion section.

Thank you for bringing this to our attention. We found your comments extremely helpful and have revised them accordingly. The revised text can be read as follows:

“The results indicate that both inputs work comparably well in classifying emotions. This finding is demonstrated by the fact that the best valence performance was obtained using a 1-D ECG, and the best arousal performance was acquired using a 2-D ECG. Additionally, ERS with 1-D ECG was combined with dimensionality reduction, called LDA. The presence of LDA improved the ERS performance in valence emotion but not in arousal. In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.

However, it is worth mentioning that the results obtained using 2-D ECG demonstrated potential for use as an input modality for the ERS. Additionally, 2-D ECGs are appealing because the format enables the use of a variety of image-based methods such as image augmentation to increase the data size, convolution neural networks (CNN), and the application of transfer learning from models trained using large data. To summarise, the ERS performance of the two ECG inputs is comparable since both yield a promising outcome for emotion recognition.”
The paper titled “Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system” looks interesting. Although it is a short paper, I found it interesting and am positive about this work, but it should go through some modification:

We appreciate your feedback and suggestions. We have revised the manuscript as needed.

Figure 2 should be redrawn, only providing the reference is not enough to present a figure or image in the article.

Thank you for your suggestions. As per your recommendations, we have redrawn the figure and an additional note has been added in the legend. As several modifications were made to the manuscript, Figure 2 was originally changed to Figure 1. This figure can be found on page 4.

The ECG signals used in this work contain some baseline wandering. It should be removed before further analysis. A simple technique with MATLAB code is described in Rahman et al (2019)1 to remove baseline wander. I am expecting a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal.

Thank you for your suggestion. We did use the method suggested as cited accordingly. However, your expectation of having a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal is inappropriate for our work because the focus of this paper is to compare emotion classification performance using 1-D and 2-D ECGs to investigate the effect of the ECG input format on the ERS.

In addition to that, is it possible to use LSTM to the ECG time series to find the accuracy of the emotion recognition from ECG signals?

As far as we know, it is possible to use LSTM on the ECG time series to find the accuracy of emotion recognition from ECG signals. The work by Song et al. used LSTM for the same purpose as your concern. Below is the paper, which you can take a look at. We hope it answers your question.

T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, and Z. Cui, “MPED: A multi-modal physiological emotion database for discrete emotion recognition,” IEEE Access, vol. 7, no. October, pp. 12177–12191, 2019.

Please provide detailed information about transforming the signal to image conversion.

Thank you for pointing this out. We have revised the 2-D ECG subsection, where the pre-processing process until the transformation from 1-D to 2-D is explained in detail. This subsection can be found on pages 11 and 12, which can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

The reference-related comparison and discussion should be completed in the result and discussions. If possible, avoid referencing in the Conclusion section.

Thank you for bringing this to our attention. We found your comments extremely helpful and have revised them accordingly. The revised text can be read as follows:

“The results indicate that both inputs work comparably well in classifying emotions. This finding is demonstrated by the fact that the best valence performance was obtained using a 1-D ECG, and the best arousal performance was acquired using a 2-D ECG. Additionally, ERS with 1-D ECG was combined with dimensionality reduction, called LDA. The presence of LDA improved the ERS performance in valence emotion but not in arousal. In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.

However, it is worth mentioning that the results obtained using 2-D ECG demonstrated potential for use as an input modality for the ERS. Additionally, 2-D ECGs are appealing because the format enables the use of a variety of image-based methods such as image augmentation to increase the data size, convolution neural networks (CNN), and the application of transfer learning from models trained using large data. To summarise, the ERS performance of the two ECG inputs is comparable since both yield a promising outcome for emotion recognition.”
Competing Interests: The authors declare that they have no conflict of interest. Close
Report a concern

Views

Reviewer Report 16 Mar 2022

Marios Fanourakis, University of Geneva, Geneva, Switzerland

Not Approved

https://doi.org/10.5256/f1000research.76896.r126091

Is the work clearly and accurately presented and does it cite the current literature?

No
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

References

1. He W, Ye Y, Pan T, Meng Q, et al.: Emotion Recognition from ECG Signals Contaminated by Motion Artifacts,. International Conference on Intelligent Technology and Embedded Systems (ICITES), 2021. 2021. 125-130 Publisher Full Text
2. Sarkar P, Etemad A: Self-supervised ECG Representation Learning for Emotion Recognition. IEEE Transactions on Affective Computing. 2021. Publisher Full Text
3. Hasnul MA, Ab Aziz NA, Aziz AA: Evaluation of TEAP and AuBT as ECG's Feature Extraction Toolbox for Emotion Recognition System. 2021 IEEE 9th Conference on Systems, Process and Control (ICSPC 2021), 2021. 52-57 Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Affective computing, emotion recognition

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

CITE

Report a concern

Author Response 30 May 2022

Sharifah Noor Masidayu Sayed Ismail, Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, 75450, Malaysia

30 May 2022

Author Response

The authors use the DREAMER dataset to compare the emotion recognition performance of features extracted from the time-series ECG signal versus features extracted from images of the ECG wave signal. ... Continue reading The authors use the DREAMER dataset to compare the emotion recognition performance of features extracted from the time-series ECG signal versus features extracted from images of the ECG wave signal. An SVM model is used as the classifier.

Overall, the structure of the report is not coherent, the related work is incomplete, and the motivation for this work is not convincing. Furthermore, many important details are missing about the dataset and the methods used which makes it difficult to trust the results and the comparisons they make with other works.

Thanks for taking the time to review our manuscript. We have revised the manuscript based on your comments and suggestions accordingly.

Structure: several improvements to be made, information seems to be scattered throughout the article. For example, information about the dataset is present in both the introduction and methods sections. It is best to keep this information in the same section. Same for the related works.

Thanks for pointing this out. We agree with your suggestion and have attempted to address the issues. Therefore, we have revised each section as per suggestions by revising the information written in the manuscript accordingly. The revised part can be found on pages 3 until 13.

Authors should give a brief explanation on what are ECG wave images in the introduction, otherwise readers in the emotion recognition field might confuse them with spectrograms which are more widely used in the field. It may also be better to change the term "numerical data" to "time-series data" or "1D data" (2D being an image).

Thank you for drawing our attention to this. As you mentioned, we have added an explanation of what ECG wave images are in the Introduction section that reads as follows:

“Fundamentally, ECG is used to measure electrical activity in the human heart by attaching electrodes to the human body. Due to the continual blood pumping action to the body, the electrical activity of the heart can be found in the sinoatrial node. The electrocardiogram signal is composed of three basic components: P, QRS, and T waves (Figure 1). P waves are produced during atrium depolarization, QRS complexes are produced during ventricular depolarization, and T waves are produced during ventricle recovery.

Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

Additionally, we also added Figure 2 that shows the snippet of the 2-D ECG from the PDF file. This figure can be found on page 4.

Furthermore, as per suggestion, we have changed the terms "numerical data" to "1D ECG" and "wave images" to "2-D ECG".

Typo on page 3: EGC instead of ECG.

Thank you so much for catching these glaring and confusing errors, which we have now corrected.

From the references pertaining to the use of ECG images (1-6, 23,24) only half actually use ECG wave images (2,4,23,24). The rest either use time-series or convert the time-series to spectrograms. The ones that do use ECG images, mainly analyze individual beats and not the entire ECG wave in order to detect medical heart issues. For the emotion recognition use-case it is necessary to analyze significantly larger regions of the signal than individual beats.

Based on your comment, we have revised our Related Works section and corrected them accordingly. Additionally, we added the summary of existing works that use 1-D and 2-D ECG input with their purposes, tabulated in Table 1. The revision of these issues can be found on page 4 until 8.

In emotion recognition, it is common to transform the ECG signal into a spectrogram image. The authors do not cite any work mentioning this method.

Thanks for pointing this out. We agree with your comments. Therefore, we have revised our Related Works section and provided improvements through the fourth paragraph on page 6:
.
“Despite rising popularity among medical practitioners in assessing patients' cardiac disease, 2-D ECG remains inadequate compared to 1-D ECG usage as a type of input in emotion recognition studies. As a result, the number of studies employing 1-D ECG in ERS is higher than that utilizing 2-D ECG in ERS. However, rather than employing a printout-based 2-D ECG, emotion researchers classified human emotions using 2-D ECG spectral images. For example, ¹⁵ determines the R-peaks of the electrocardiogram prior to generating the R-R interval (RRI) spectrogram. Following that, CNN was used to classify the emotions, with an accuracy rate greater than 90%. Elalamy et al. ³⁰ used ResNet-50 to extract features from a 2-D ECG spectrogram. Then, Logistic Regression (LR) was employed as a classifier and achieved an accuracy of 78.30% in classifying emotions.”

Additionally, in Table 1, the ECG input was listed as either 1-D or 2-D ECGs, where 2-D was further categorised into standard 2-D ECG or spectral 2-D ECG.

Authors do not include any references to other works that use the DREAMER dataset ECG signals for emotion recognition, here are some:

Wenwen He et al. 2021 Emotion Recognition from ECG Signals Contaminated by Motion Artifacts

Pritam Sarkar et al. 2020 Self-supervised ECG Representation Learning for Emotion Recognition

I also came across another publication of some of the co-authors which would be advantageous to reference:

Muhammad Anas Hasnul et al. 2021 Evaluation of TEAP and AuBT as ECG’s Feature Extraction Toolbox for Emotion Recognition System.

Thank you for the paper suggested. We have added the reference according to your recommendation, which you can find on page 6 as follows:

“Additionally, numerous other researchers also used the ECG signals from the DREAMER dataset to perform emotion recognition. For instance, 1-D ECG data from the DREAMER dataset is utilized by Wenwen He et al. ²⁴ that suggested an approach for emotion recognition using ECG contaminated by motion artefacts. The proposed approach improved classification accuracy by 5% to 15%. Additionally, Pritam and Ali ²⁵ also employed 1-D ECG from the DREAMER dataset to develop the self-supervised deep multi-task learning framework ERS, which consists of two stages of learning: ECG representation learning and emotion classification learning. The accuracy gained in this study was greater than 70%. Hasnul et al. ¹² also used the 1-D ECG by DREAMER dataset to compare the performance of two feature extractor toolboxes. They noted that the dataset's size and the type of emotion classified might affect the suitability of the extracted features.”

In the Emotion model paragraph (page 3): it is still unclear which model of emotions was used, Ekman or Russell? Authors say "the latter" (referring to Russel) but then mention binary classification which may be confusing since the arousal/valence space is continuous making it a regression problem or at least multiclass and not binary. Authors should re-word this part to make it clear.

Thank you for pointing this out. The reviewer is correct; the original phrase is confusing. Therefore, we have removed this part to avoid any further confusion. Furthermore, this concern has been changed and revised to make it more straightforward and understandable. The revised text was located in the experimental setting subsection under the Method section on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Referring to wearable ECG devices, the authors state: "However, most of these devices store the ECG as images instead of raw numerical data". No references or other market analysis is provided to show that this is the case. The use of ECG wave images for emotion recognition is not properly motivated. I fail to see any advantage of using ECG wave images over the time-series data unless ECG time-series data is not available.

Thank you for drawing our attention to this, and we agree with the comments. Therefore, we have cited the necessary references and revised the statement as per the suggestion. The revised text can be found on page 4 as follows:

“Previous research on human emotions has primarily relied on either direct analysis of 1-D data ^{12 – 14} or the conversion of 1-D data to a 2-D spectral image ¹⁵ prior to identifying the emotions. Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

How was the data annotated in the DREAMER dataset? Were they continuous annotations or a single annotation per video clip? This is very relevant to include in the article. A few more words about the dataset are needed like a short description of the experimental protocol.

Thank you for bringing this issue to our attention. The short description of the DREAMER dataset has been improved to address this issue. The revised text was located in the Method section under a subsection called "The dataset (DREAMER") on page 8:

“This study used ECG signals from Katsigiannis and Ramzan ¹³ called DREAMER. The DREAMER dataset is a freely accessible database of electroencephalogram (EEG) and electrocardiogram (ECG) signals used in emotion research. However, EEG signals were removed from this study because the primary focus is on ECG signals. The ECG was recorded using the SHIMMER ECG sensor at 256 Hz and stored in 1-D format. The DREAMER dataset contains 414 ECG recordings from 23 subjects who were exposed to 18 audio-visual stimuli designed to evoke emotion. Each participant assessed their emotions on a scale of 1 to 5 for arousal, valence, and dominance. However, because this study was primarily concerned with arousal and valence ratings, participants' evaluations of dominance were discarded.”

Arousal/valence rating values are ranging from 1 to 5 in the DREAMER dataset. The authors never explain how they split them into two classes (high/low) for the binary classification.

Thank you for bringing this to our attention. We have improved this part by adding an explanation of how we categorise emotions into two classes. The explanation can be found on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Authors do not include sufficient information on how the time-series ECG data was converted to an image (resolution, compression, windowing), or how the data was treated in general. Did the authors use the entire ECG signal for each video? Was there any windowing? Since the authors did not properly summarize the dataset (how were the videos annotated?) it is difficult to grasp or guess on how the data was processed.

We agree with this comment. Therefore, we improved the explanation part of pre-processing the 2-D ECG. The improvised text can be found in the first and second paragraphs of subsection 2-D ECG on page 11.

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

None of the cited literature used any of the image feature extraction methods that the authors used, and the authors did not discuss their reasoning for why they selected those image feature extraction methods and not the ones established in the ECG image analysis literature that they cited. Some more illustrations of these features may be useful besides the one in Figure 5 in order to convince the readers.

Thank you for pointing this out. Cite literature either using their own algorithm to detect peaks on PQRST waves or automatically extracting ECG features using a deep learning system. However, we had included our reason for employing these image feature extraction methods, which we believed could add a valuable contribution to the state-of-the-art. Additionally, we removed Figure 5 and replaced it with the description of each feature extractor to help convince the readers. The revised text can be found on pages 11 and 12:

“The area of interest for 2-D ECG is laying on the PQRST waves, making the peaks detector the best approach to be employed. Therefore, six different feature extractors that could extract peaks, edges, or corners were applied to extract features from 2-D ECGs using Python version 3.8.5:
1. ORB ³⁷: ORB features are invariant to rotation and noise because they are a combination of Features from Accelerated Segment Test (FAST) detection and Binary Robust Independent Elementary Features (BRIEF) description methods.
2. SIFT ³⁸: SIFT identifies feature points by searching for local maxima on the images using Difference-of-Gaussians (DoG) operators. The description approach generates a 16x16 neighbourhood around each identified feature and sub-blocks the region. SIFT is also rotation and scale invariant.
3. KAZE ³⁹: KAZE is based on the scale of the normalised determinant of the Hessian Matrix, with the maxima of detector responses being captured as feature points using a moving window. Additionally, KAZE makes use of non-linear space via non-linear diffusion filtering to reduce noise while keeping the borders of regions in images.
4. AKAZE ⁴⁰: AKAZE is a more sophisticated version of KAZE that is based on the Hessian Matrix determinant. Scharr filters were employed to enhance the quality of the invariance rotation, rendering AKAZE features rotation- and scale-invariant.
5. BRISK ⁴¹: While searching for maxima in the scale-space pyramid, BRISK detects corners using the AGAST algorithm and filters them using the FAST Corner Score. Additionally, the BRISK description is based on the recognised characteristic direction of each feature, which is necessary for rotation invariance.
6. HOG ⁴²: HOG is a feature descriptor that is used to compute the gradient value for each pixel. The image shape denoted the edge or gradient structure derived using a high-quality local gradient intensity distribution.”

Support vector machine: It is not clear what preprocessing steps were applied to the data. For example, in the DREAMER dataset baseline they only use the last 60s of data for each film clip.

Thank you for this comment. The pre-processing part can be found in the Methods section under the 1-D ECG and 2-D ECG subsections. The pre-processing for 1-D ECG can be read as follows:

“The AUBT and TEAP feature extractors were included with the Low Pass Filter (LPF), a filter meant to reject all undesirable frequencies in a signal. The LPF was one of the most widely used filters before the computation of statistical features for physiological signals ^{31, 32}. As a result, automated 1-D ECG pre-processing utilizing LPF was performed in this study to reduce muscle and respiratory noise in ECG signals.”

Whereas the pre-processing for 2-D ECG can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG.”

Data was divided 80:20 and 10-fold cross-validation was used. The authors do not specify exactly how the data was split (see DREAMER dataset paper section V as an example). From reading this part, I can only assume that all data from all participants was used for each fold (general model), something that is diverging from how the data was split in the DREAMER dataset paper (they made models for each individual participant). Therefore, any comparisons to the results of the DREAMER baseline are invalid.

Thank you for pointing this out. We have addressed this issue by adding a new subsection under Method, namely, Experimental Setting. This subsection is located on pages 13 and 14, and can be read as follows:

“The hyperparameters for SVM were tuned using an exhaustive parameter search tool, GridSearchCV, from Scikit-learn that automates the tuning procedure ⁴⁵. This study tuned only the parameters with a high and relative tuning risk and left the remainder at their default values because they are the least sensitive to the hyperparameter tuning process, as suggested by Weerts, Mueller, and Vanschoren ⁴⁶.

The dataset was split into a reasonable proportion of training and testing sets to evaluate the model's performance on new unseen data. This study used a stratified train-test split of 80:20 for the training and testing sets. This strategy guarantees the dataset's exact proportions of samples in each class are preserved.

Additionally, as we had a small dataset size, this study applied KFold Cross-Validation, with the number of folds set to 10, the most commonly used number in prior research to improve ERS performance.”

Additionally, the summary of the experimental setting values has been tabulated in Table 5 on page 14.

Results: If the classes are unbalanced (as the DREAMER dataset paper indicates) accuracy is not valid on its own, include f1 score, and/or Cohen's Kappa..

Thank you for your suggestions. As per your recommendations, we have included the F1-score along with the accuracy to address the unbalanced class distribution issue. This result can be found in Table 6 on page 14.

Discussion: "Ref. 11" has nothing to do with the statement in that paragraph. LDA was actually applied in the DREAMER dataset paper and they reported that there were no significant differences in performance.

Thank you for your great observations. We have revised that part as needed. Regarding the LDA, we agree with you. However, as far as we know, LDA was used for classification purposes in the DREAMER dataset paper but not for dimension reduction.
The authors use the DREAMER dataset to compare the emotion recognition performance of features extracted from the time-series ECG signal versus features extracted from images of the ECG wave signal. An SVM model is used as the classifier.

Overall, the structure of the report is not coherent, the related work is incomplete, and the motivation for this work is not convincing. Furthermore, many important details are missing about the dataset and the methods used which makes it difficult to trust the results and the comparisons they make with other works.

Thanks for taking the time to review our manuscript. We have revised the manuscript based on your comments and suggestions accordingly.

Structure: several improvements to be made, information seems to be scattered throughout the article. For example, information about the dataset is present in both the introduction and methods sections. It is best to keep this information in the same section. Same for the related works.

Thanks for pointing this out. We agree with your suggestion and have attempted to address the issues. Therefore, we have revised each section as per suggestions by revising the information written in the manuscript accordingly. The revised part can be found on pages 3 until 13.

Authors should give a brief explanation on what are ECG wave images in the introduction, otherwise readers in the emotion recognition field might confuse them with spectrograms which are more widely used in the field. It may also be better to change the term "numerical data" to "time-series data" or "1D data" (2D being an image).

Thank you for drawing our attention to this. As you mentioned, we have added an explanation of what ECG wave images are in the Introduction section that reads as follows:

“Fundamentally, ECG is used to measure electrical activity in the human heart by attaching electrodes to the human body. Due to the continual blood pumping action to the body, the electrical activity of the heart can be found in the sinoatrial node. The electrocardiogram signal is composed of three basic components: P, QRS, and T waves (Figure 1). P waves are produced during atrium depolarization, QRS complexes are produced during ventricular depolarization, and T waves are produced during ventricle recovery.

Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

Additionally, we also added Figure 2 that shows the snippet of the 2-D ECG from the PDF file. This figure can be found on page 4.

Furthermore, as per suggestion, we have changed the terms "numerical data" to "1D ECG" and "wave images" to "2-D ECG".

Typo on page 3: EGC instead of ECG.

Thank you so much for catching these glaring and confusing errors, which we have now corrected.

From the references pertaining to the use of ECG images (1-6, 23,24) only half actually use ECG wave images (2,4,23,24). The rest either use time-series or convert the time-series to spectrograms. The ones that do use ECG images, mainly analyze individual beats and not the entire ECG wave in order to detect medical heart issues. For the emotion recognition use-case it is necessary to analyze significantly larger regions of the signal than individual beats.

Based on your comment, we have revised our Related Works section and corrected them accordingly. Additionally, we added the summary of existing works that use 1-D and 2-D ECG input with their purposes, tabulated in Table 1. The revision of these issues can be found on page 4 until 8.

In emotion recognition, it is common to transform the ECG signal into a spectrogram image. The authors do not cite any work mentioning this method.

Thanks for pointing this out. We agree with your comments. Therefore, we have revised our Related Works section and provided improvements through the fourth paragraph on page 6:
.
“Despite rising popularity among medical practitioners in assessing patients' cardiac disease, 2-D ECG remains inadequate compared to 1-D ECG usage as a type of input in emotion recognition studies. As a result, the number of studies employing 1-D ECG in ERS is higher than that utilizing 2-D ECG in ERS. However, rather than employing a printout-based 2-D ECG, emotion researchers classified human emotions using 2-D ECG spectral images. For example, ¹⁵ determines the R-peaks of the electrocardiogram prior to generating the R-R interval (RRI) spectrogram. Following that, CNN was used to classify the emotions, with an accuracy rate greater than 90%. Elalamy et al. ³⁰ used ResNet-50 to extract features from a 2-D ECG spectrogram. Then, Logistic Regression (LR) was employed as a classifier and achieved an accuracy of 78.30% in classifying emotions.”

Additionally, in Table 1, the ECG input was listed as either 1-D or 2-D ECGs, where 2-D was further categorised into standard 2-D ECG or spectral 2-D ECG.

Authors do not include any references to other works that use the DREAMER dataset ECG signals for emotion recognition, here are some:

Wenwen He et al. 2021 Emotion Recognition from ECG Signals Contaminated by Motion Artifacts

Pritam Sarkar et al. 2020 Self-supervised ECG Representation Learning for Emotion Recognition

I also came across another publication of some of the co-authors which would be advantageous to reference:

Muhammad Anas Hasnul et al. 2021 Evaluation of TEAP and AuBT as ECG’s Feature Extraction Toolbox for Emotion Recognition System.

Thank you for the paper suggested. We have added the reference according to your recommendation, which you can find on page 6 as follows:

“Additionally, numerous other researchers also used the ECG signals from the DREAMER dataset to perform emotion recognition. For instance, 1-D ECG data from the DREAMER dataset is utilized by Wenwen He et al. ²⁴ that suggested an approach for emotion recognition using ECG contaminated by motion artefacts. The proposed approach improved classification accuracy by 5% to 15%. Additionally, Pritam and Ali ²⁵ also employed 1-D ECG from the DREAMER dataset to develop the self-supervised deep multi-task learning framework ERS, which consists of two stages of learning: ECG representation learning and emotion classification learning. The accuracy gained in this study was greater than 70%. Hasnul et al. ¹² also used the 1-D ECG by DREAMER dataset to compare the performance of two feature extractor toolboxes. They noted that the dataset's size and the type of emotion classified might affect the suitability of the extracted features.”

In the Emotion model paragraph (page 3): it is still unclear which model of emotions was used, Ekman or Russell? Authors say "the latter" (referring to Russel) but then mention binary classification which may be confusing since the arousal/valence space is continuous making it a regression problem or at least multiclass and not binary. Authors should re-word this part to make it clear.

Thank you for pointing this out. The reviewer is correct; the original phrase is confusing. Therefore, we have removed this part to avoid any further confusion. Furthermore, this concern has been changed and revised to make it more straightforward and understandable. The revised text was located in the experimental setting subsection under the Method section on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Referring to wearable ECG devices, the authors state: "However, most of these devices store the ECG as images instead of raw numerical data". No references or other market analysis is provided to show that this is the case. The use of ECG wave images for emotion recognition is not properly motivated. I fail to see any advantage of using ECG wave images over the time-series data unless ECG time-series data is not available.

Thank you for drawing our attention to this, and we agree with the comments. Therefore, we have cited the necessary references and revised the statement as per the suggestion. The revised text can be found on page 4 as follows:

“Previous research on human emotions has primarily relied on either direct analysis of 1-D data ^{12 – 14} or the conversion of 1-D data to a 2-D spectral image ¹⁵ prior to identifying the emotions. Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

How was the data annotated in the DREAMER dataset? Were they continuous annotations or a single annotation per video clip? This is very relevant to include in the article. A few more words about the dataset are needed like a short description of the experimental protocol.

Thank you for bringing this issue to our attention. The short description of the DREAMER dataset has been improved to address this issue. The revised text was located in the Method section under a subsection called "The dataset (DREAMER") on page 8:

“This study used ECG signals from Katsigiannis and Ramzan ¹³ called DREAMER. The DREAMER dataset is a freely accessible database of electroencephalogram (EEG) and electrocardiogram (ECG) signals used in emotion research. However, EEG signals were removed from this study because the primary focus is on ECG signals. The ECG was recorded using the SHIMMER ECG sensor at 256 Hz and stored in 1-D format. The DREAMER dataset contains 414 ECG recordings from 23 subjects who were exposed to 18 audio-visual stimuli designed to evoke emotion. Each participant assessed their emotions on a scale of 1 to 5 for arousal, valence, and dominance. However, because this study was primarily concerned with arousal and valence ratings, participants' evaluations of dominance were discarded.”

Arousal/valence rating values are ranging from 1 to 5 in the DREAMER dataset. The authors never explain how they split them into two classes (high/low) for the binary classification.

Thank you for bringing this to our attention. We have improved this part by adding an explanation of how we categorise emotions into two classes. The explanation can be found on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Authors do not include sufficient information on how the time-series ECG data was converted to an image (resolution, compression, windowing), or how the data was treated in general. Did the authors use the entire ECG signal for each video? Was there any windowing? Since the authors did not properly summarize the dataset (how were the videos annotated?) it is difficult to grasp or guess on how the data was processed.

We agree with this comment. Therefore, we improved the explanation part of pre-processing the 2-D ECG. The improvised text can be found in the first and second paragraphs of subsection 2-D ECG on page 11.

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

None of the cited literature used any of the image feature extraction methods that the authors used, and the authors did not discuss their reasoning for why they selected those image feature extraction methods and not the ones established in the ECG image analysis literature that they cited. Some more illustrations of these features may be useful besides the one in Figure 5 in order to convince the readers.

Thank you for pointing this out. Cite literature either using their own algorithm to detect peaks on PQRST waves or automatically extracting ECG features using a deep learning system. However, we had included our reason for employing these image feature extraction methods, which we believed could add a valuable contribution to the state-of-the-art. Additionally, we removed Figure 5 and replaced it with the description of each feature extractor to help convince the readers. The revised text can be found on pages 11 and 12:

“The area of interest for 2-D ECG is laying on the PQRST waves, making the peaks detector the best approach to be employed. Therefore, six different feature extractors that could extract peaks, edges, or corners were applied to extract features from 2-D ECGs using Python version 3.8.5:
1. ORB ³⁷: ORB features are invariant to rotation and noise because they are a combination of Features from Accelerated Segment Test (FAST) detection and Binary Robust Independent Elementary Features (BRIEF) description methods.
2. SIFT ³⁸: SIFT identifies feature points by searching for local maxima on the images using Difference-of-Gaussians (DoG) operators. The description approach generates a 16x16 neighbourhood around each identified feature and sub-blocks the region. SIFT is also rotation and scale invariant.
3. KAZE ³⁹: KAZE is based on the scale of the normalised determinant of the Hessian Matrix, with the maxima of detector responses being captured as feature points using a moving window. Additionally, KAZE makes use of non-linear space via non-linear diffusion filtering to reduce noise while keeping the borders of regions in images.
4. AKAZE ⁴⁰: AKAZE is a more sophisticated version of KAZE that is based on the Hessian Matrix determinant. Scharr filters were employed to enhance the quality of the invariance rotation, rendering AKAZE features rotation- and scale-invariant.
5. BRISK ⁴¹: While searching for maxima in the scale-space pyramid, BRISK detects corners using the AGAST algorithm and filters them using the FAST Corner Score. Additionally, the BRISK description is based on the recognised characteristic direction of each feature, which is necessary for rotation invariance.
6. HOG ⁴²: HOG is a feature descriptor that is used to compute the gradient value for each pixel. The image shape denoted the edge or gradient structure derived using a high-quality local gradient intensity distribution.”

Support vector machine: It is not clear what preprocessing steps were applied to the data. For example, in the DREAMER dataset baseline they only use the last 60s of data for each film clip.

Thank you for this comment. The pre-processing part can be found in the Methods section under the 1-D ECG and 2-D ECG subsections. The pre-processing for 1-D ECG can be read as follows:

“The AUBT and TEAP feature extractors were included with the Low Pass Filter (LPF), a filter meant to reject all undesirable frequencies in a signal. The LPF was one of the most widely used filters before the computation of statistical features for physiological signals ^{31, 32}. As a result, automated 1-D ECG pre-processing utilizing LPF was performed in this study to reduce muscle and respiratory noise in ECG signals.”

Whereas the pre-processing for 2-D ECG can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG.”

Data was divided 80:20 and 10-fold cross-validation was used. The authors do not specify exactly how the data was split (see DREAMER dataset paper section V as an example). From reading this part, I can only assume that all data from all participants was used for each fold (general model), something that is diverging from how the data was split in the DREAMER dataset paper (they made models for each individual participant). Therefore, any comparisons to the results of the DREAMER baseline are invalid.

Thank you for pointing this out. We have addressed this issue by adding a new subsection under Method, namely, Experimental Setting. This subsection is located on pages 13 and 14, and can be read as follows:

“The hyperparameters for SVM were tuned using an exhaustive parameter search tool, GridSearchCV, from Scikit-learn that automates the tuning procedure ⁴⁵. This study tuned only the parameters with a high and relative tuning risk and left the remainder at their default values because they are the least sensitive to the hyperparameter tuning process, as suggested by Weerts, Mueller, and Vanschoren ⁴⁶.

The dataset was split into a reasonable proportion of training and testing sets to evaluate the model's performance on new unseen data. This study used a stratified train-test split of 80:20 for the training and testing sets. This strategy guarantees the dataset's exact proportions of samples in each class are preserved.

Additionally, as we had a small dataset size, this study applied KFold Cross-Validation, with the number of folds set to 10, the most commonly used number in prior research to improve ERS performance.”

Additionally, the summary of the experimental setting values has been tabulated in Table 5 on page 14.

Results: If the classes are unbalanced (as the DREAMER dataset paper indicates) accuracy is not valid on its own, include f1 score, and/or Cohen's Kappa..

Thank you for your suggestions. As per your recommendations, we have included the F1-score along with the accuracy to address the unbalanced class distribution issue. This result can be found in Table 6 on page 14.

Discussion: "Ref. 11" has nothing to do with the statement in that paragraph. LDA was actually applied in the DREAMER dataset paper and they reported that there were no significant differences in performance.

Thank you for your great observations. We have revised that part as needed. Regarding the LDA, we agree with you. However, as far as we know, LDA was used for classification purposes in the DREAMER dataset paper but not for dimension reduction.
Competing Interests: The authors declare that they have no conflict of interest. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 30 May 2022

Sharifah Noor Masidayu Sayed Ismail, Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, 75450, Malaysia

30 May 2022

Author Response

The authors use the DREAMER dataset to compare the emotion recognition performance of features extracted from the time-series ECG signal versus features extracted from images of the ECG wave signal. ... Continue reading The authors use the DREAMER dataset to compare the emotion recognition performance of features extracted from the time-series ECG signal versus features extracted from images of the ECG wave signal. An SVM model is used as the classifier.

Overall, the structure of the report is not coherent, the related work is incomplete, and the motivation for this work is not convincing. Furthermore, many important details are missing about the dataset and the methods used which makes it difficult to trust the results and the comparisons they make with other works.

Thanks for taking the time to review our manuscript. We have revised the manuscript based on your comments and suggestions accordingly.

Structure: several improvements to be made, information seems to be scattered throughout the article. For example, information about the dataset is present in both the introduction and methods sections. It is best to keep this information in the same section. Same for the related works.

Thanks for pointing this out. We agree with your suggestion and have attempted to address the issues. Therefore, we have revised each section as per suggestions by revising the information written in the manuscript accordingly. The revised part can be found on pages 3 until 13.

Authors should give a brief explanation on what are ECG wave images in the introduction, otherwise readers in the emotion recognition field might confuse them with spectrograms which are more widely used in the field. It may also be better to change the term "numerical data" to "time-series data" or "1D data" (2D being an image).

Thank you for drawing our attention to this. As you mentioned, we have added an explanation of what ECG wave images are in the Introduction section that reads as follows:

“Fundamentally, ECG is used to measure electrical activity in the human heart by attaching electrodes to the human body. Due to the continual blood pumping action to the body, the electrical activity of the heart can be found in the sinoatrial node. The electrocardiogram signal is composed of three basic components: P, QRS, and T waves (Figure 1). P waves are produced during atrium depolarization, QRS complexes are produced during ventricular depolarization, and T waves are produced during ventricle recovery.

Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

Additionally, we also added Figure 2 that shows the snippet of the 2-D ECG from the PDF file. This figure can be found on page 4.

Furthermore, as per suggestion, we have changed the terms "numerical data" to "1D ECG" and "wave images" to "2-D ECG".

Typo on page 3: EGC instead of ECG.

Thank you so much for catching these glaring and confusing errors, which we have now corrected.

From the references pertaining to the use of ECG images (1-6, 23,24) only half actually use ECG wave images (2,4,23,24). The rest either use time-series or convert the time-series to spectrograms. The ones that do use ECG images, mainly analyze individual beats and not the entire ECG wave in order to detect medical heart issues. For the emotion recognition use-case it is necessary to analyze significantly larger regions of the signal than individual beats.

Based on your comment, we have revised our Related Works section and corrected them accordingly. Additionally, we added the summary of existing works that use 1-D and 2-D ECG input with their purposes, tabulated in Table 1. The revision of these issues can be found on page 4 until 8.

In emotion recognition, it is common to transform the ECG signal into a spectrogram image. The authors do not cite any work mentioning this method.

Thanks for pointing this out. We agree with your comments. Therefore, we have revised our Related Works section and provided improvements through the fourth paragraph on page 6:
.
“Despite rising popularity among medical practitioners in assessing patients' cardiac disease, 2-D ECG remains inadequate compared to 1-D ECG usage as a type of input in emotion recognition studies. As a result, the number of studies employing 1-D ECG in ERS is higher than that utilizing 2-D ECG in ERS. However, rather than employing a printout-based 2-D ECG, emotion researchers classified human emotions using 2-D ECG spectral images. For example, ¹⁵ determines the R-peaks of the electrocardiogram prior to generating the R-R interval (RRI) spectrogram. Following that, CNN was used to classify the emotions, with an accuracy rate greater than 90%. Elalamy et al. ³⁰ used ResNet-50 to extract features from a 2-D ECG spectrogram. Then, Logistic Regression (LR) was employed as a classifier and achieved an accuracy of 78.30% in classifying emotions.”

Additionally, in Table 1, the ECG input was listed as either 1-D or 2-D ECGs, where 2-D was further categorised into standard 2-D ECG or spectral 2-D ECG.

Authors do not include any references to other works that use the DREAMER dataset ECG signals for emotion recognition, here are some:

Wenwen He et al. 2021 Emotion Recognition from ECG Signals Contaminated by Motion Artifacts

Pritam Sarkar et al. 2020 Self-supervised ECG Representation Learning for Emotion Recognition

I also came across another publication of some of the co-authors which would be advantageous to reference:

Muhammad Anas Hasnul et al. 2021 Evaluation of TEAP and AuBT as ECG’s Feature Extraction Toolbox for Emotion Recognition System.

Thank you for the paper suggested. We have added the reference according to your recommendation, which you can find on page 6 as follows:

“Additionally, numerous other researchers also used the ECG signals from the DREAMER dataset to perform emotion recognition. For instance, 1-D ECG data from the DREAMER dataset is utilized by Wenwen He et al. ²⁴ that suggested an approach for emotion recognition using ECG contaminated by motion artefacts. The proposed approach improved classification accuracy by 5% to 15%. Additionally, Pritam and Ali ²⁵ also employed 1-D ECG from the DREAMER dataset to develop the self-supervised deep multi-task learning framework ERS, which consists of two stages of learning: ECG representation learning and emotion classification learning. The accuracy gained in this study was greater than 70%. Hasnul et al. ¹² also used the 1-D ECG by DREAMER dataset to compare the performance of two feature extractor toolboxes. They noted that the dataset's size and the type of emotion classified might affect the suitability of the extracted features.”

In the Emotion model paragraph (page 3): it is still unclear which model of emotions was used, Ekman or Russell? Authors say "the latter" (referring to Russel) but then mention binary classification which may be confusing since the arousal/valence space is continuous making it a regression problem or at least multiclass and not binary. Authors should re-word this part to make it clear.

Thank you for pointing this out. The reviewer is correct; the original phrase is confusing. Therefore, we have removed this part to avoid any further confusion. Furthermore, this concern has been changed and revised to make it more straightforward and understandable. The revised text was located in the experimental setting subsection under the Method section on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Referring to wearable ECG devices, the authors state: "However, most of these devices store the ECG as images instead of raw numerical data". No references or other market analysis is provided to show that this is the case. The use of ECG wave images for emotion recognition is not properly motivated. I fail to see any advantage of using ECG wave images over the time-series data unless ECG time-series data is not available.

Thank you for drawing our attention to this, and we agree with the comments. Therefore, we have cited the necessary references and revised the statement as per the suggestion. The revised text can be found on page 4 as follows:

“Previous research on human emotions has primarily relied on either direct analysis of 1-D data ^{12 – 14} or the conversion of 1-D data to a 2-D spectral image ¹⁵ prior to identifying the emotions. Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

How was the data annotated in the DREAMER dataset? Were they continuous annotations or a single annotation per video clip? This is very relevant to include in the article. A few more words about the dataset are needed like a short description of the experimental protocol.

Thank you for bringing this issue to our attention. The short description of the DREAMER dataset has been improved to address this issue. The revised text was located in the Method section under a subsection called "The dataset (DREAMER") on page 8:

“This study used ECG signals from Katsigiannis and Ramzan ¹³ called DREAMER. The DREAMER dataset is a freely accessible database of electroencephalogram (EEG) and electrocardiogram (ECG) signals used in emotion research. However, EEG signals were removed from this study because the primary focus is on ECG signals. The ECG was recorded using the SHIMMER ECG sensor at 256 Hz and stored in 1-D format. The DREAMER dataset contains 414 ECG recordings from 23 subjects who were exposed to 18 audio-visual stimuli designed to evoke emotion. Each participant assessed their emotions on a scale of 1 to 5 for arousal, valence, and dominance. However, because this study was primarily concerned with arousal and valence ratings, participants' evaluations of dominance were discarded.”

Arousal/valence rating values are ranging from 1 to 5 in the DREAMER dataset. The authors never explain how they split them into two classes (high/low) for the binary classification.

Thank you for bringing this to our attention. We have improved this part by adding an explanation of how we categorise emotions into two classes. The explanation can be found on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Authors do not include sufficient information on how the time-series ECG data was converted to an image (resolution, compression, windowing), or how the data was treated in general. Did the authors use the entire ECG signal for each video? Was there any windowing? Since the authors did not properly summarize the dataset (how were the videos annotated?) it is difficult to grasp or guess on how the data was processed.

We agree with this comment. Therefore, we improved the explanation part of pre-processing the 2-D ECG. The improvised text can be found in the first and second paragraphs of subsection 2-D ECG on page 11.

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

None of the cited literature used any of the image feature extraction methods that the authors used, and the authors did not discuss their reasoning for why they selected those image feature extraction methods and not the ones established in the ECG image analysis literature that they cited. Some more illustrations of these features may be useful besides the one in Figure 5 in order to convince the readers.

Thank you for pointing this out. Cite literature either using their own algorithm to detect peaks on PQRST waves or automatically extracting ECG features using a deep learning system. However, we had included our reason for employing these image feature extraction methods, which we believed could add a valuable contribution to the state-of-the-art. Additionally, we removed Figure 5 and replaced it with the description of each feature extractor to help convince the readers. The revised text can be found on pages 11 and 12:

“The area of interest for 2-D ECG is laying on the PQRST waves, making the peaks detector the best approach to be employed. Therefore, six different feature extractors that could extract peaks, edges, or corners were applied to extract features from 2-D ECGs using Python version 3.8.5:
1. ORB ³⁷: ORB features are invariant to rotation and noise because they are a combination of Features from Accelerated Segment Test (FAST) detection and Binary Robust Independent Elementary Features (BRIEF) description methods.
2. SIFT ³⁸: SIFT identifies feature points by searching for local maxima on the images using Difference-of-Gaussians (DoG) operators. The description approach generates a 16x16 neighbourhood around each identified feature and sub-blocks the region. SIFT is also rotation and scale invariant.
3. KAZE ³⁹: KAZE is based on the scale of the normalised determinant of the Hessian Matrix, with the maxima of detector responses being captured as feature points using a moving window. Additionally, KAZE makes use of non-linear space via non-linear diffusion filtering to reduce noise while keeping the borders of regions in images.
4. AKAZE ⁴⁰: AKAZE is a more sophisticated version of KAZE that is based on the Hessian Matrix determinant. Scharr filters were employed to enhance the quality of the invariance rotation, rendering AKAZE features rotation- and scale-invariant.
5. BRISK ⁴¹: While searching for maxima in the scale-space pyramid, BRISK detects corners using the AGAST algorithm and filters them using the FAST Corner Score. Additionally, the BRISK description is based on the recognised characteristic direction of each feature, which is necessary for rotation invariance.
6. HOG ⁴²: HOG is a feature descriptor that is used to compute the gradient value for each pixel. The image shape denoted the edge or gradient structure derived using a high-quality local gradient intensity distribution.”

Support vector machine: It is not clear what preprocessing steps were applied to the data. For example, in the DREAMER dataset baseline they only use the last 60s of data for each film clip.

Thank you for this comment. The pre-processing part can be found in the Methods section under the 1-D ECG and 2-D ECG subsections. The pre-processing for 1-D ECG can be read as follows:

“The AUBT and TEAP feature extractors were included with the Low Pass Filter (LPF), a filter meant to reject all undesirable frequencies in a signal. The LPF was one of the most widely used filters before the computation of statistical features for physiological signals ^{31, 32}. As a result, automated 1-D ECG pre-processing utilizing LPF was performed in this study to reduce muscle and respiratory noise in ECG signals.”

Whereas the pre-processing for 2-D ECG can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG.”

Data was divided 80:20 and 10-fold cross-validation was used. The authors do not specify exactly how the data was split (see DREAMER dataset paper section V as an example). From reading this part, I can only assume that all data from all participants was used for each fold (general model), something that is diverging from how the data was split in the DREAMER dataset paper (they made models for each individual participant). Therefore, any comparisons to the results of the DREAMER baseline are invalid.

Thank you for pointing this out. We have addressed this issue by adding a new subsection under Method, namely, Experimental Setting. This subsection is located on pages 13 and 14, and can be read as follows:

“The hyperparameters for SVM were tuned using an exhaustive parameter search tool, GridSearchCV, from Scikit-learn that automates the tuning procedure ⁴⁵. This study tuned only the parameters with a high and relative tuning risk and left the remainder at their default values because they are the least sensitive to the hyperparameter tuning process, as suggested by Weerts, Mueller, and Vanschoren ⁴⁶.

The dataset was split into a reasonable proportion of training and testing sets to evaluate the model's performance on new unseen data. This study used a stratified train-test split of 80:20 for the training and testing sets. This strategy guarantees the dataset's exact proportions of samples in each class are preserved.

Additionally, as we had a small dataset size, this study applied KFold Cross-Validation, with the number of folds set to 10, the most commonly used number in prior research to improve ERS performance.”

Additionally, the summary of the experimental setting values has been tabulated in Table 5 on page 14.

Results: If the classes are unbalanced (as the DREAMER dataset paper indicates) accuracy is not valid on its own, include f1 score, and/or Cohen's Kappa..

Thank you for your suggestions. As per your recommendations, we have included the F1-score along with the accuracy to address the unbalanced class distribution issue. This result can be found in Table 6 on page 14.

Discussion: "Ref. 11" has nothing to do with the statement in that paragraph. LDA was actually applied in the DREAMER dataset paper and they reported that there were no significant differences in performance.

Thank you for your great observations. We have revised that part as needed. Regarding the LDA, we agree with you. However, as far as we know, LDA was used for classification purposes in the DREAMER dataset paper but not for dimension reduction.
The authors use the DREAMER dataset to compare the emotion recognition performance of features extracted from the time-series ECG signal versus features extracted from images of the ECG wave signal. An SVM model is used as the classifier.

Overall, the structure of the report is not coherent, the related work is incomplete, and the motivation for this work is not convincing. Furthermore, many important details are missing about the dataset and the methods used which makes it difficult to trust the results and the comparisons they make with other works.

Thanks for taking the time to review our manuscript. We have revised the manuscript based on your comments and suggestions accordingly.

Structure: several improvements to be made, information seems to be scattered throughout the article. For example, information about the dataset is present in both the introduction and methods sections. It is best to keep this information in the same section. Same for the related works.

Thanks for pointing this out. We agree with your suggestion and have attempted to address the issues. Therefore, we have revised each section as per suggestions by revising the information written in the manuscript accordingly. The revised part can be found on pages 3 until 13.

Authors should give a brief explanation on what are ECG wave images in the introduction, otherwise readers in the emotion recognition field might confuse them with spectrograms which are more widely used in the field. It may also be better to change the term "numerical data" to "time-series data" or "1D data" (2D being an image).

Thank you for drawing our attention to this. As you mentioned, we have added an explanation of what ECG wave images are in the Introduction section that reads as follows:

“Fundamentally, ECG is used to measure electrical activity in the human heart by attaching electrodes to the human body. Due to the continual blood pumping action to the body, the electrical activity of the heart can be found in the sinoatrial node. The electrocardiogram signal is composed of three basic components: P, QRS, and T waves (Figure 1). P waves are produced during atrium depolarization, QRS complexes are produced during ventricular depolarization, and T waves are produced during ventricle recovery.

Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

Additionally, we also added Figure 2 that shows the snippet of the 2-D ECG from the PDF file. This figure can be found on page 4.

Furthermore, as per suggestion, we have changed the terms "numerical data" to "1D ECG" and "wave images" to "2-D ECG".

Typo on page 3: EGC instead of ECG.

Thank you so much for catching these glaring and confusing errors, which we have now corrected.

From the references pertaining to the use of ECG images (1-6, 23,24) only half actually use ECG wave images (2,4,23,24). The rest either use time-series or convert the time-series to spectrograms. The ones that do use ECG images, mainly analyze individual beats and not the entire ECG wave in order to detect medical heart issues. For the emotion recognition use-case it is necessary to analyze significantly larger regions of the signal than individual beats.

Based on your comment, we have revised our Related Works section and corrected them accordingly. Additionally, we added the summary of existing works that use 1-D and 2-D ECG input with their purposes, tabulated in Table 1. The revision of these issues can be found on page 4 until 8.

In emotion recognition, it is common to transform the ECG signal into a spectrogram image. The authors do not cite any work mentioning this method.

Thanks for pointing this out. We agree with your comments. Therefore, we have revised our Related Works section and provided improvements through the fourth paragraph on page 6:
.
“Despite rising popularity among medical practitioners in assessing patients' cardiac disease, 2-D ECG remains inadequate compared to 1-D ECG usage as a type of input in emotion recognition studies. As a result, the number of studies employing 1-D ECG in ERS is higher than that utilizing 2-D ECG in ERS. However, rather than employing a printout-based 2-D ECG, emotion researchers classified human emotions using 2-D ECG spectral images. For example, ¹⁵ determines the R-peaks of the electrocardiogram prior to generating the R-R interval (RRI) spectrogram. Following that, CNN was used to classify the emotions, with an accuracy rate greater than 90%. Elalamy et al. ³⁰ used ResNet-50 to extract features from a 2-D ECG spectrogram. Then, Logistic Regression (LR) was employed as a classifier and achieved an accuracy of 78.30% in classifying emotions.”

Additionally, in Table 1, the ECG input was listed as either 1-D or 2-D ECGs, where 2-D was further categorised into standard 2-D ECG or spectral 2-D ECG.

Authors do not include any references to other works that use the DREAMER dataset ECG signals for emotion recognition, here are some:

Wenwen He et al. 2021 Emotion Recognition from ECG Signals Contaminated by Motion Artifacts

Pritam Sarkar et al. 2020 Self-supervised ECG Representation Learning for Emotion Recognition

I also came across another publication of some of the co-authors which would be advantageous to reference:

Muhammad Anas Hasnul et al. 2021 Evaluation of TEAP and AuBT as ECG’s Feature Extraction Toolbox for Emotion Recognition System.

Thank you for the paper suggested. We have added the reference according to your recommendation, which you can find on page 6 as follows:

“Additionally, numerous other researchers also used the ECG signals from the DREAMER dataset to perform emotion recognition. For instance, 1-D ECG data from the DREAMER dataset is utilized by Wenwen He et al. ²⁴ that suggested an approach for emotion recognition using ECG contaminated by motion artefacts. The proposed approach improved classification accuracy by 5% to 15%. Additionally, Pritam and Ali ²⁵ also employed 1-D ECG from the DREAMER dataset to develop the self-supervised deep multi-task learning framework ERS, which consists of two stages of learning: ECG representation learning and emotion classification learning. The accuracy gained in this study was greater than 70%. Hasnul et al. ¹² also used the 1-D ECG by DREAMER dataset to compare the performance of two feature extractor toolboxes. They noted that the dataset's size and the type of emotion classified might affect the suitability of the extracted features.”

In the Emotion model paragraph (page 3): it is still unclear which model of emotions was used, Ekman or Russell? Authors say "the latter" (referring to Russel) but then mention binary classification which may be confusing since the arousal/valence space is continuous making it a regression problem or at least multiclass and not binary. Authors should re-word this part to make it clear.

Thank you for pointing this out. The reviewer is correct; the original phrase is confusing. Therefore, we have removed this part to avoid any further confusion. Furthermore, this concern has been changed and revised to make it more straightforward and understandable. The revised text was located in the experimental setting subsection under the Method section on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Referring to wearable ECG devices, the authors state: "However, most of these devices store the ECG as images instead of raw numerical data". No references or other market analysis is provided to show that this is the case. The use of ECG wave images for emotion recognition is not properly motivated. I fail to see any advantage of using ECG wave images over the time-series data unless ECG time-series data is not available.

Thank you for drawing our attention to this, and we agree with the comments. Therefore, we have cited the necessary references and revised the statement as per the suggestion. The revised text can be found on page 4 as follows:

“Previous research on human emotions has primarily relied on either direct analysis of 1-D data ^{12 – 14} or the conversion of 1-D data to a 2-D spectral image ¹⁵ prior to identifying the emotions. Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

How was the data annotated in the DREAMER dataset? Were they continuous annotations or a single annotation per video clip? This is very relevant to include in the article. A few more words about the dataset are needed like a short description of the experimental protocol.

Thank you for bringing this issue to our attention. The short description of the DREAMER dataset has been improved to address this issue. The revised text was located in the Method section under a subsection called "The dataset (DREAMER") on page 8:

“This study used ECG signals from Katsigiannis and Ramzan ¹³ called DREAMER. The DREAMER dataset is a freely accessible database of electroencephalogram (EEG) and electrocardiogram (ECG) signals used in emotion research. However, EEG signals were removed from this study because the primary focus is on ECG signals. The ECG was recorded using the SHIMMER ECG sensor at 256 Hz and stored in 1-D format. The DREAMER dataset contains 414 ECG recordings from 23 subjects who were exposed to 18 audio-visual stimuli designed to evoke emotion. Each participant assessed their emotions on a scale of 1 to 5 for arousal, valence, and dominance. However, because this study was primarily concerned with arousal and valence ratings, participants' evaluations of dominance were discarded.”

Arousal/valence rating values are ranging from 1 to 5 in the DREAMER dataset. The authors never explain how they split them into two classes (high/low) for the binary classification.

Thank you for bringing this to our attention. We have improved this part by adding an explanation of how we categorise emotions into two classes. The explanation can be found on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Authors do not include sufficient information on how the time-series ECG data was converted to an image (resolution, compression, windowing), or how the data was treated in general. Did the authors use the entire ECG signal for each video? Was there any windowing? Since the authors did not properly summarize the dataset (how were the videos annotated?) it is difficult to grasp or guess on how the data was processed.

We agree with this comment. Therefore, we improved the explanation part of pre-processing the 2-D ECG. The improvised text can be found in the first and second paragraphs of subsection 2-D ECG on page 11.

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

None of the cited literature used any of the image feature extraction methods that the authors used, and the authors did not discuss their reasoning for why they selected those image feature extraction methods and not the ones established in the ECG image analysis literature that they cited. Some more illustrations of these features may be useful besides the one in Figure 5 in order to convince the readers.

Thank you for pointing this out. Cite literature either using their own algorithm to detect peaks on PQRST waves or automatically extracting ECG features using a deep learning system. However, we had included our reason for employing these image feature extraction methods, which we believed could add a valuable contribution to the state-of-the-art. Additionally, we removed Figure 5 and replaced it with the description of each feature extractor to help convince the readers. The revised text can be found on pages 11 and 12:

“The area of interest for 2-D ECG is laying on the PQRST waves, making the peaks detector the best approach to be employed. Therefore, six different feature extractors that could extract peaks, edges, or corners were applied to extract features from 2-D ECGs using Python version 3.8.5:
1. ORB ³⁷: ORB features are invariant to rotation and noise because they are a combination of Features from Accelerated Segment Test (FAST) detection and Binary Robust Independent Elementary Features (BRIEF) description methods.
2. SIFT ³⁸: SIFT identifies feature points by searching for local maxima on the images using Difference-of-Gaussians (DoG) operators. The description approach generates a 16x16 neighbourhood around each identified feature and sub-blocks the region. SIFT is also rotation and scale invariant.
3. KAZE ³⁹: KAZE is based on the scale of the normalised determinant of the Hessian Matrix, with the maxima of detector responses being captured as feature points using a moving window. Additionally, KAZE makes use of non-linear space via non-linear diffusion filtering to reduce noise while keeping the borders of regions in images.
4. AKAZE ⁴⁰: AKAZE is a more sophisticated version of KAZE that is based on the Hessian Matrix determinant. Scharr filters were employed to enhance the quality of the invariance rotation, rendering AKAZE features rotation- and scale-invariant.
5. BRISK ⁴¹: While searching for maxima in the scale-space pyramid, BRISK detects corners using the AGAST algorithm and filters them using the FAST Corner Score. Additionally, the BRISK description is based on the recognised characteristic direction of each feature, which is necessary for rotation invariance.
6. HOG ⁴²: HOG is a feature descriptor that is used to compute the gradient value for each pixel. The image shape denoted the edge or gradient structure derived using a high-quality local gradient intensity distribution.”

Support vector machine: It is not clear what preprocessing steps were applied to the data. For example, in the DREAMER dataset baseline they only use the last 60s of data for each film clip.

Thank you for this comment. The pre-processing part can be found in the Methods section under the 1-D ECG and 2-D ECG subsections. The pre-processing for 1-D ECG can be read as follows:

“The AUBT and TEAP feature extractors were included with the Low Pass Filter (LPF), a filter meant to reject all undesirable frequencies in a signal. The LPF was one of the most widely used filters before the computation of statistical features for physiological signals ^{31, 32}. As a result, automated 1-D ECG pre-processing utilizing LPF was performed in this study to reduce muscle and respiratory noise in ECG signals.”

Whereas the pre-processing for 2-D ECG can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG.”

Data was divided 80:20 and 10-fold cross-validation was used. The authors do not specify exactly how the data was split (see DREAMER dataset paper section V as an example). From reading this part, I can only assume that all data from all participants was used for each fold (general model), something that is diverging from how the data was split in the DREAMER dataset paper (they made models for each individual participant). Therefore, any comparisons to the results of the DREAMER baseline are invalid.

Thank you for pointing this out. We have addressed this issue by adding a new subsection under Method, namely, Experimental Setting. This subsection is located on pages 13 and 14, and can be read as follows:

“The hyperparameters for SVM were tuned using an exhaustive parameter search tool, GridSearchCV, from Scikit-learn that automates the tuning procedure ⁴⁵. This study tuned only the parameters with a high and relative tuning risk and left the remainder at their default values because they are the least sensitive to the hyperparameter tuning process, as suggested by Weerts, Mueller, and Vanschoren ⁴⁶.

The dataset was split into a reasonable proportion of training and testing sets to evaluate the model's performance on new unseen data. This study used a stratified train-test split of 80:20 for the training and testing sets. This strategy guarantees the dataset's exact proportions of samples in each class are preserved.

Additionally, as we had a small dataset size, this study applied KFold Cross-Validation, with the number of folds set to 10, the most commonly used number in prior research to improve ERS performance.”

Additionally, the summary of the experimental setting values has been tabulated in Table 5 on page 14.

Results: If the classes are unbalanced (as the DREAMER dataset paper indicates) accuracy is not valid on its own, include f1 score, and/or Cohen's Kappa..

Thank you for your suggestions. As per your recommendations, we have included the F1-score along with the accuracy to address the unbalanced class distribution issue. This result can be found in Table 6 on page 14.

Discussion: "Ref. 11" has nothing to do with the statement in that paragraph. LDA was actually applied in the DREAMER dataset paper and they reported that there were no significant differences in performance.

Thank you for your great observations. We have revised that part as needed. Regarding the LDA, we agree with you. However, as far as we know, LDA was used for classification purposes in the DREAMER dataset paper but not for dimension reduction.
Competing Interests: The authors declare that they have no conflict of interest. Close
Report a concern

Views

Reviewer Report 21 Feb 2022

Umesh Chandra Pati, Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela, Odisha, India

Approved with Reservations

https://doi.org/10.5256/f1000research.76896.r121060

Literature survey is poor. More number of related state-of-the-art works should be cited. There is no citation of the work taking into account both ECG image and ECG numerical data.
It has been mentioned that the converted ECG images have been resized to 60% of the original size to reduce the computational time. What is the reason for choosing 60%?
There is no analysis of the computational complexity of the proposed method.
In Table 4, emotion classification accuracies for both ECG image and ECG numerical data have been provided. Can these accuracy values be accepted in practical applications?
There is no comparison of the proposed method with state-of-the-art methods.
References should be complete in all respect and in a uniform style. A few exceptions are 1, 32, 39, 40.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Computer Vision, Image/Video Processing, IoT, Artificial Intelligence.

CITE

Report a concern

Author Response 30 May 2022

Sharifah Noor Masidayu Sayed Ismail, Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, 75450, Malaysia

30 May 2022

Author Response

The paper presents a comparison of the Emotion Recognition System (ERS) performance using ECG image and ECG numerical data. DREAMER dataset containing 23 ECG recordings has been used for experimentation. ... Continue reading The paper presents a comparison of the Emotion Recognition System (ERS) performance using ECG image and ECG numerical data. DREAMER dataset containing 23 ECG recordings has been used for experimentation. ECG numerical data has been converted to ECG images for comparison. The proposed method consists of feature extraction, dimension reduction, and emotion classification. Arousal and Valence accuracy have been compared using different feature extractors on ECG image and numerical data. ORB, SIFT, KAZE, AKAZE, BRISK, and HOG have been used to extract features from ECG image data whereas TEAP and AUBT have been used to extract features from ECG numerical data. It has been concluded that ECG numerical data provides better performance of emotion classification in comparison to ECG image data.

The authors have done a good job in addressing a typical issue. However, there are many aspects that need attention. Hence, the following suggestions should be addressed to enhance the quality of the manuscript.

Thank you for taking the time to review our manuscript. We reviewed the comments and made improvements based on the comments provided accordingly.

Literature survey is poor. More number of related state-of-the-art works should be cited. There is no citation of the work taking into account both ECG image and ECG numerical data.

We have revised Section Related Works as per the suggestion by adding more related state-of-the-art work citations. The revised part is on pages 5 until 8. We also include the summary information about the existing works that employed 1-D and 2-D ECG input, which is tabulated in Table 2 on pages 6 through 8. Furthermore, the work that considers both ECG images and ECG numerical data is also included in this section. This work can be found on page 6, which reads as follows:

"Additionally, Mandal et al. ⁵ published a study comparing 1-D and 2-D ECGs for the diagnosis of Ventricular Arrhythmia. They concluded that both ECG inputs are effective at detecting the disease."

It has been mentioned that the converted ECG images have been resized to 60% of the original size to reduce the computational time. What is the reason for choosing 60%?

Thank you for drawing our attention to this. We have improved the explanation of this matter in the manuscript. You can find the text on page 11 that reads as follows:

“Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford.”

There is no analysis of the computational complexity of the proposed method.

Thank you for pointing out this matter. We have included the analysis of the computational complexity under the result section, tabulated in Table 7, which discussed the time analysis of both inputs used to train and test the ERS model. The analysis was located on page 14:

“For comparison purposes, the computation time for both ECG inputs was recorded and reported in Table 7. The average time required to compute 1-D is 1.58 ± 0.07 seconds. In comparison, the average computation time for 2-D is 3377.425 ± 3138.875 seconds. Therefore, according to the observation, 2-D took the longest computation time, whereas 1-D obtained the shortest.”

Additionally, the analysis of the computational time was touched on in the Discussion and Conclusion section as follows:

“In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.”

In Table 4, emotion classification accuracies for both ECG image and ECG numerical data have been provided. Can these accuracy values be accepted in practical applications?

Thank you for pointing this out. For your information, our results have been updated according to the result of the latest experiment. This result is tabulated in Table 6 on page 13. Based on this result, the accuracy and F1-score achieved for this study are on par with the existing works, which shows that our ERS model can be accepted in practical applications. However, this does not rule out the possibility of improving this result, as there is much more room to improve the ERS performance to develop a more robust ERS in the future.

There is no comparison of the proposed method with state-of-the-art methods.

A comparison with existing work (DREAMER) has been provided for 1-D ECG. This comparison can be found in the result section on page 15, tabulated in Table 6. However, no existing work using the 2-D ECG of the DREAMER dataset has been reported. Therefore, no comparison can be made.

References should be complete in all respect and in a uniform style. A few exceptions are 1, 32, 39, 40.

Thank you for bringing our attention to this. We have revised the references and citation part accordingly and completed it in all respects and in a uniform style as per suggestions.
The paper presents a comparison of the Emotion Recognition System (ERS) performance using ECG image and ECG numerical data. DREAMER dataset containing 23 ECG recordings has been used for experimentation. ECG numerical data has been converted to ECG images for comparison. The proposed method consists of feature extraction, dimension reduction, and emotion classification. Arousal and Valence accuracy have been compared using different feature extractors on ECG image and numerical data. ORB, SIFT, KAZE, AKAZE, BRISK, and HOG have been used to extract features from ECG image data whereas TEAP and AUBT have been used to extract features from ECG numerical data. It has been concluded that ECG numerical data provides better performance of emotion classification in comparison to ECG image data.

The authors have done a good job in addressing a typical issue. However, there are many aspects that need attention. Hence, the following suggestions should be addressed to enhance the quality of the manuscript.

Thank you for taking the time to review our manuscript. We reviewed the comments and made improvements based on the comments provided accordingly.

Literature survey is poor. More number of related state-of-the-art works should be cited. There is no citation of the work taking into account both ECG image and ECG numerical data.

We have revised Section Related Works as per the suggestion by adding more related state-of-the-art work citations. The revised part is on pages 5 until 8. We also include the summary information about the existing works that employed 1-D and 2-D ECG input, which is tabulated in Table 2 on pages 6 through 8. Furthermore, the work that considers both ECG images and ECG numerical data is also included in this section. This work can be found on page 6, which reads as follows:

"Additionally, Mandal et al. ⁵ published a study comparing 1-D and 2-D ECGs for the diagnosis of Ventricular Arrhythmia. They concluded that both ECG inputs are effective at detecting the disease."

It has been mentioned that the converted ECG images have been resized to 60% of the original size to reduce the computational time. What is the reason for choosing 60%?

Thank you for drawing our attention to this. We have improved the explanation of this matter in the manuscript. You can find the text on page 11 that reads as follows:

“Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford.”

There is no analysis of the computational complexity of the proposed method.

Thank you for pointing out this matter. We have included the analysis of the computational complexity under the result section, tabulated in Table 7, which discussed the time analysis of both inputs used to train and test the ERS model. The analysis was located on page 14:

“For comparison purposes, the computation time for both ECG inputs was recorded and reported in Table 7. The average time required to compute 1-D is 1.58 ± 0.07 seconds. In comparison, the average computation time for 2-D is 3377.425 ± 3138.875 seconds. Therefore, according to the observation, 2-D took the longest computation time, whereas 1-D obtained the shortest.”

Additionally, the analysis of the computational time was touched on in the Discussion and Conclusion section as follows:

“In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.”

In Table 4, emotion classification accuracies for both ECG image and ECG numerical data have been provided. Can these accuracy values be accepted in practical applications?

Thank you for pointing this out. For your information, our results have been updated according to the result of the latest experiment. This result is tabulated in Table 6 on page 13. Based on this result, the accuracy and F1-score achieved for this study are on par with the existing works, which shows that our ERS model can be accepted in practical applications. However, this does not rule out the possibility of improving this result, as there is much more room to improve the ERS performance to develop a more robust ERS in the future.

There is no comparison of the proposed method with state-of-the-art methods.

A comparison with existing work (DREAMER) has been provided for 1-D ECG. This comparison can be found in the result section on page 15, tabulated in Table 6. However, no existing work using the 2-D ECG of the DREAMER dataset has been reported. Therefore, no comparison can be made.

References should be complete in all respect and in a uniform style. A few exceptions are 1, 32, 39, 40.

Thank you for bringing our attention to this. We have revised the references and citation part accordingly and completed it in all respects and in a uniform style as per suggestions.
Competing Interests: The authors declare that they have no conflict of interest. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 30 May 2022

Sharifah Noor Masidayu Sayed Ismail, Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, 75450, Malaysia

30 May 2022

Author Response

The paper presents a comparison of the Emotion Recognition System (ERS) performance using ECG image and ECG numerical data. DREAMER dataset containing 23 ECG recordings has been used for experimentation. ... Continue reading The paper presents a comparison of the Emotion Recognition System (ERS) performance using ECG image and ECG numerical data. DREAMER dataset containing 23 ECG recordings has been used for experimentation. ECG numerical data has been converted to ECG images for comparison. The proposed method consists of feature extraction, dimension reduction, and emotion classification. Arousal and Valence accuracy have been compared using different feature extractors on ECG image and numerical data. ORB, SIFT, KAZE, AKAZE, BRISK, and HOG have been used to extract features from ECG image data whereas TEAP and AUBT have been used to extract features from ECG numerical data. It has been concluded that ECG numerical data provides better performance of emotion classification in comparison to ECG image data.

The authors have done a good job in addressing a typical issue. However, there are many aspects that need attention. Hence, the following suggestions should be addressed to enhance the quality of the manuscript.

Thank you for taking the time to review our manuscript. We reviewed the comments and made improvements based on the comments provided accordingly.

Literature survey is poor. More number of related state-of-the-art works should be cited. There is no citation of the work taking into account both ECG image and ECG numerical data.

We have revised Section Related Works as per the suggestion by adding more related state-of-the-art work citations. The revised part is on pages 5 until 8. We also include the summary information about the existing works that employed 1-D and 2-D ECG input, which is tabulated in Table 2 on pages 6 through 8. Furthermore, the work that considers both ECG images and ECG numerical data is also included in this section. This work can be found on page 6, which reads as follows:

"Additionally, Mandal et al. ⁵ published a study comparing 1-D and 2-D ECGs for the diagnosis of Ventricular Arrhythmia. They concluded that both ECG inputs are effective at detecting the disease."

It has been mentioned that the converted ECG images have been resized to 60% of the original size to reduce the computational time. What is the reason for choosing 60%?

Thank you for drawing our attention to this. We have improved the explanation of this matter in the manuscript. You can find the text on page 11 that reads as follows:

“Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford.”

There is no analysis of the computational complexity of the proposed method.

Thank you for pointing out this matter. We have included the analysis of the computational complexity under the result section, tabulated in Table 7, which discussed the time analysis of both inputs used to train and test the ERS model. The analysis was located on page 14:

“For comparison purposes, the computation time for both ECG inputs was recorded and reported in Table 7. The average time required to compute 1-D is 1.58 ± 0.07 seconds. In comparison, the average computation time for 2-D is 3377.425 ± 3138.875 seconds. Therefore, according to the observation, 2-D took the longest computation time, whereas 1-D obtained the shortest.”

Additionally, the analysis of the computational time was touched on in the Discussion and Conclusion section as follows:

“In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.”

In Table 4, emotion classification accuracies for both ECG image and ECG numerical data have been provided. Can these accuracy values be accepted in practical applications?

Thank you for pointing this out. For your information, our results have been updated according to the result of the latest experiment. This result is tabulated in Table 6 on page 13. Based on this result, the accuracy and F1-score achieved for this study are on par with the existing works, which shows that our ERS model can be accepted in practical applications. However, this does not rule out the possibility of improving this result, as there is much more room to improve the ERS performance to develop a more robust ERS in the future.

There is no comparison of the proposed method with state-of-the-art methods.

A comparison with existing work (DREAMER) has been provided for 1-D ECG. This comparison can be found in the result section on page 15, tabulated in Table 6. However, no existing work using the 2-D ECG of the DREAMER dataset has been reported. Therefore, no comparison can be made.

References should be complete in all respect and in a uniform style. A few exceptions are 1, 32, 39, 40.

Thank you for bringing our attention to this. We have revised the references and citation part accordingly and completed it in all respects and in a uniform style as per suggestions.
The paper presents a comparison of the Emotion Recognition System (ERS) performance using ECG image and ECG numerical data. DREAMER dataset containing 23 ECG recordings has been used for experimentation. ECG numerical data has been converted to ECG images for comparison. The proposed method consists of feature extraction, dimension reduction, and emotion classification. Arousal and Valence accuracy have been compared using different feature extractors on ECG image and numerical data. ORB, SIFT, KAZE, AKAZE, BRISK, and HOG have been used to extract features from ECG image data whereas TEAP and AUBT have been used to extract features from ECG numerical data. It has been concluded that ECG numerical data provides better performance of emotion classification in comparison to ECG image data.

The authors have done a good job in addressing a typical issue. However, there are many aspects that need attention. Hence, the following suggestions should be addressed to enhance the quality of the manuscript.

Thank you for taking the time to review our manuscript. We reviewed the comments and made improvements based on the comments provided accordingly.

Literature survey is poor. More number of related state-of-the-art works should be cited. There is no citation of the work taking into account both ECG image and ECG numerical data.

We have revised Section Related Works as per the suggestion by adding more related state-of-the-art work citations. The revised part is on pages 5 until 8. We also include the summary information about the existing works that employed 1-D and 2-D ECG input, which is tabulated in Table 2 on pages 6 through 8. Furthermore, the work that considers both ECG images and ECG numerical data is also included in this section. This work can be found on page 6, which reads as follows:

"Additionally, Mandal et al. ⁵ published a study comparing 1-D and 2-D ECGs for the diagnosis of Ventricular Arrhythmia. They concluded that both ECG inputs are effective at detecting the disease."

It has been mentioned that the converted ECG images have been resized to 60% of the original size to reduce the computational time. What is the reason for choosing 60%?

Thank you for drawing our attention to this. We have improved the explanation of this matter in the manuscript. You can find the text on page 11 that reads as follows:

“Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford.”

There is no analysis of the computational complexity of the proposed method.

Thank you for pointing out this matter. We have included the analysis of the computational complexity under the result section, tabulated in Table 7, which discussed the time analysis of both inputs used to train and test the ERS model. The analysis was located on page 14:

“For comparison purposes, the computation time for both ECG inputs was recorded and reported in Table 7. The average time required to compute 1-D is 1.58 ± 0.07 seconds. In comparison, the average computation time for 2-D is 3377.425 ± 3138.875 seconds. Therefore, according to the observation, 2-D took the longest computation time, whereas 1-D obtained the shortest.”

Additionally, the analysis of the computational time was touched on in the Discussion and Conclusion section as follows:

“In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.”

In Table 4, emotion classification accuracies for both ECG image and ECG numerical data have been provided. Can these accuracy values be accepted in practical applications?

Thank you for pointing this out. For your information, our results have been updated according to the result of the latest experiment. This result is tabulated in Table 6 on page 13. Based on this result, the accuracy and F1-score achieved for this study are on par with the existing works, which shows that our ERS model can be accepted in practical applications. However, this does not rule out the possibility of improving this result, as there is much more room to improve the ERS performance to develop a more robust ERS in the future.

There is no comparison of the proposed method with state-of-the-art methods.

A comparison with existing work (DREAMER) has been provided for 1-D ECG. This comparison can be found in the result section on page 15, tabulated in Table 6. However, no existing work using the 2-D ECG of the DREAMER dataset has been reported. Therefore, no comparison can be made.

References should be complete in all respect and in a uniform style. A few exceptions are 1, 32, 39, 40.

Thank you for bringing our attention to this. We have revised the references and citation part accordingly and completed it in all respects and in a uniform style as per suggestions.
Competing Interests: The authors declare that they have no conflict of interest. Close
Report a concern

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 04 Nov 2021

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3
Version 2 (revision) 30 May 22		read	read
Version 1 04 Nov 21	read	read	read

Umesh Chandra Pati, National Institute of Technology, Rourkela, India
Marios Fanourakis, University of Geneva, Geneva, Switzerland
Md. Asadur Rahman, Military Institute of Science and Technology (MIST), Dhaka, Bangladesh

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

10 Views

06 Jun 2022 | for Version 2

Marios Fanourakis, University of Geneva, Geneva, Switzerland

10 Views Cite this report Responses(0)

Approved

After evaluating the author's revisions and replies to my previous comments I would like to upgrade the status to Approved.

The authors have carefully addressed the issues previously identified with only some minor clarifications needed.

What method was used to resize/rescale the 2D ECG images (which python function)? I see in the github code that the following method was used "cv2.resize(img, dim, interpolation = cv2.INTER_AREA)", please explain what the interpolation method cv2.INTER_AREA is.

As mentioned the 2D image is 1920x620 pixels, a 60% rescaling doesn't result in either 224x224 nor 299x299. Some information is still missing here. Furthermore, why is 224x224 and 299x299 a "standard size"? Is it that these sizes are common in other ML models? Other reasons?

When splitting the data between the training and test set, was an individual participant's data also split between the train and test sets? For example, suppose there is a participant "A", would one find some data of participant "A" in the training set and some other data of the same participant "A" in the test set, or was it split such that if there was data of participant "A" in the training set there was no data from participant "A" in the test set and vice versa? This could help to further interpret the results.

From a quick glance at the github code, it seems that participants' data could be present in both training and test sets. This is not inherently bad and I don't believe it invalidates the results and conclusions presented, especially since the work's main contribution is to compare the performance of 1D ECG signals to 2D ECG images as inputs to ML models. Nonetheless, it is important to mention.

A short definition of "stratified train-test split" could be helpful for readers.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Affective computing, emotion recognition

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

10 Views

31 May 2022 | for Version 2

Md. Asadur Rahman, Department of Biomedical Engineering, Military Institute of Science and Technology (MIST), Dhaka, Bangladesh

10 Views Cite this report Responses(0)

Approved

The revised version can be approved. I am satisfied so far.

Competing Interests

No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

16 Views

16 Mar 2022 | for Version 1

Md. Asadur Rahman, Department of Biomedical Engineering, Military Institute of Science and Technology (MIST), Dhaka, Bangladesh

16 Views Cite this report Responses(1)

Approved With Reservations

Figure 2 should be redrawn, only providing the reference is not enough to present a figure or image in the article.
The ECG signals used in this work contain some baseline wandering. It should be removed before further analysis. A simple technique with Matlab code is described in Rahman et al (2019)¹ to remove baseline wander. I am expecting a result comparing the emotion recognition rate before and after the baseline wander removal from the ECG signal.
In addition to that, is it possible to use LSTM to the ECG time series to find the accuracy of the emotion recognition from ECG signals?
Please provide detailed information about transforming the signal to image conversion.
The reference-related comparison and discussion should be completed in the result and discussions. If possible, avoid referencing in the Conclusion section.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Biomedical Signal Processing

Respond to this report

Responses (1)

Author Response

30 May 2022

Sharifah Noor Masidayu Sayed Ismail, Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, 75450, Malaysia

The paper titled “Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system” looks interesting. Although it is a short paper, I found it interesting and am positive about this work, but it should go through some modification:

We appreciate your feedback and suggestions. We have revised the manuscript as needed.

Figure 2 should be redrawn, only providing the reference is not enough to present a figure or image in the article.

Thank you for your suggestions. As per your recommendations, we have redrawn the figure and an additional note has been added in the legend. As several modifications were made to the manuscript, Figure 2 was originally changed to Figure 1. This figure can be found on page 4.

The ECG signals used in this work contain some baseline wandering. It should be removed before further analysis. A simple technique with MATLAB code is described in Rahman et al (2019)1 to remove baseline wander. I am expecting a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal.

Thank you for your suggestion. We did use the method suggested as cited accordingly. However, your expectation of having a result comparing the emotion recognition before and after the baseline wander removal from the ECG signal is inappropriate for our work because the focus of this paper is to compare emotion classification performance using 1-D and 2-D ECGs to investigate the effect of the ECG input format on the ERS.

In addition to that, is it possible to use LSTM to the ECG time series to find the accuracy of the emotion recognition from ECG signals?

As far as we know, it is possible to use LSTM on the ECG time series to find the accuracy of emotion recognition from ECG signals. The work by Song et al. used LSTM for the same purpose as your concern. Below is the paper, which you can take a look at. We hope it answers your question.

T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, and Z. Cui, “MPED: A multi-modal physiological emotion database for discrete emotion recognition,” IEEE Access, vol. 7, no. October, pp. 12177–12191, 2019.

Please provide detailed information about transforming the signal to image conversion.

Thank you for pointing this out. We have revised the 2-D ECG subsection, where the pre-processing process until the transformation from 1-D to 2-D is explained in detail. This subsection can be found on pages 11 and 12, which can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

The reference-related comparison and discussion should be completed in the result and discussions. If possible, avoid referencing in the Conclusion section.

Thank you for bringing this to our attention. We found your comments extremely helpful and have revised them accordingly. The revised text can be read as follows:

“The results indicate that both inputs work comparably well in classifying emotions. This finding is demonstrated by the fact that the best valence performance was obtained using a 1-D ECG, and the best arousal performance was acquired using a 2-D ECG. Additionally, ERS with 1-D ECG was combined with dimensionality reduction, called LDA. The presence of LDA improved the ERS performance in valence emotion but not in arousal. In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.

However, it is worth mentioning that the results obtained using 2-D ECG demonstrated potential for use as an input modality for the ERS. Additionally, 2-D ECGs are appealing because the format enables the use of a variety of image-based methods such as image augmentation to increase the data size, convolution neural networks (CNN), and the application of transfer learning from models trained using large data. To summarise, the ERS performance of the two ECG inputs is comparable since both yield a promising outcome for emotion recognition.”

View more View less

Competing Interests

The authors declare that they have no conflict of interest.

Back to all reports

Reviewer Report

15 Views

16 Mar 2022 | for Version 1

Marios Fanourakis, University of Geneva, Geneva, Switzerland

15 Views Cite this report Responses(1)

Not Approved

Is the work clearly and accurately presented and does it cite the current literature?

No
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Affective computing, emotion recognition

Respond to this report

Responses (1)

Author Response

30 May 2022

Sharifah Noor Masidayu Sayed Ismail, Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, 75450, Malaysia

The authors use the DREAMER dataset to compare the emotion recognition performance of features extracted from the time-series ECG signal versus features extracted from images of the ECG wave signal. An SVM model is used as the classifier.

Overall, the structure of the report is not coherent, the related work is incomplete, and the motivation for this work is not convincing. Furthermore, many important details are missing about the dataset and the methods used which makes it difficult to trust the results and the comparisons they make with other works.

Thanks for taking the time to review our manuscript. We have revised the manuscript based on your comments and suggestions accordingly.

Structure: several improvements to be made, information seems to be scattered throughout the article. For example, information about the dataset is present in both the introduction and methods sections. It is best to keep this information in the same section. Same for the related works.

Thanks for pointing this out. We agree with your suggestion and have attempted to address the issues. Therefore, we have revised each section as per suggestions by revising the information written in the manuscript accordingly. The revised part can be found on pages 3 until 13.

Authors should give a brief explanation on what are ECG wave images in the introduction, otherwise readers in the emotion recognition field might confuse them with spectrograms which are more widely used in the field. It may also be better to change the term "numerical data" to "time-series data" or "1D data" (2D being an image).

Thank you for drawing our attention to this. As you mentioned, we have added an explanation of what ECG wave images are in the Introduction section that reads as follows:

“Fundamentally, ECG is used to measure electrical activity in the human heart by attaching electrodes to the human body. Due to the continual blood pumping action to the body, the electrical activity of the heart can be found in the sinoatrial node. The electrocardiogram signal is composed of three basic components: P, QRS, and T waves (Figure 1). P waves are produced during atrium depolarization, QRS complexes are produced during ventricular depolarization, and T waves are produced during ventricle recovery.

Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

Additionally, we also added Figure 2 that shows the snippet of the 2-D ECG from the PDF file. This figure can be found on page 4.

Furthermore, as per suggestion, we have changed the terms "numerical data" to "1D ECG" and "wave images" to "2-D ECG".

Typo on page 3: EGC instead of ECG.

Thank you so much for catching these glaring and confusing errors, which we have now corrected.

From the references pertaining to the use of ECG images (1-6, 23,24) only half actually use ECG wave images (2,4,23,24). The rest either use time-series or convert the time-series to spectrograms. The ones that do use ECG images, mainly analyze individual beats and not the entire ECG wave in order to detect medical heart issues. For the emotion recognition use-case it is necessary to analyze significantly larger regions of the signal than individual beats.

Based on your comment, we have revised our Related Works section and corrected them accordingly. Additionally, we added the summary of existing works that use 1-D and 2-D ECG input with their purposes, tabulated in Table 1. The revision of these issues can be found on page 4 until 8.

In emotion recognition, it is common to transform the ECG signal into a spectrogram image. The authors do not cite any work mentioning this method.

Thanks for pointing this out. We agree with your comments. Therefore, we have revised our Related Works section and provided improvements through the fourth paragraph on page 6:
.
“Despite rising popularity among medical practitioners in assessing patients' cardiac disease, 2-D ECG remains inadequate compared to 1-D ECG usage as a type of input in emotion recognition studies. As a result, the number of studies employing 1-D ECG in ERS is higher than that utilizing 2-D ECG in ERS. However, rather than employing a printout-based 2-D ECG, emotion researchers classified human emotions using 2-D ECG spectral images. For example, ¹⁵ determines the R-peaks of the electrocardiogram prior to generating the R-R interval (RRI) spectrogram. Following that, CNN was used to classify the emotions, with an accuracy rate greater than 90%. Elalamy et al. ³⁰ used ResNet-50 to extract features from a 2-D ECG spectrogram. Then, Logistic Regression (LR) was employed as a classifier and achieved an accuracy of 78.30% in classifying emotions.”

Additionally, in Table 1, the ECG input was listed as either 1-D or 2-D ECGs, where 2-D was further categorised into standard 2-D ECG or spectral 2-D ECG.

Authors do not include any references to other works that use the DREAMER dataset ECG signals for emotion recognition, here are some:

Wenwen He et al. 2021 Emotion Recognition from ECG Signals Contaminated by Motion Artifacts

Pritam Sarkar et al. 2020 Self-supervised ECG Representation Learning for Emotion Recognition

I also came across another publication of some of the co-authors which would be advantageous to reference:

Muhammad Anas Hasnul et al. 2021 Evaluation of TEAP and AuBT as ECG’s Feature Extraction Toolbox for Emotion Recognition System.

Thank you for the paper suggested. We have added the reference according to your recommendation, which you can find on page 6 as follows:

“Additionally, numerous other researchers also used the ECG signals from the DREAMER dataset to perform emotion recognition. For instance, 1-D ECG data from the DREAMER dataset is utilized by Wenwen He et al. ²⁴ that suggested an approach for emotion recognition using ECG contaminated by motion artefacts. The proposed approach improved classification accuracy by 5% to 15%. Additionally, Pritam and Ali ²⁵ also employed 1-D ECG from the DREAMER dataset to develop the self-supervised deep multi-task learning framework ERS, which consists of two stages of learning: ECG representation learning and emotion classification learning. The accuracy gained in this study was greater than 70%. Hasnul et al. ¹² also used the 1-D ECG by DREAMER dataset to compare the performance of two feature extractor toolboxes. They noted that the dataset's size and the type of emotion classified might affect the suitability of the extracted features.”

In the Emotion model paragraph (page 3): it is still unclear which model of emotions was used, Ekman or Russell? Authors say "the latter" (referring to Russel) but then mention binary classification which may be confusing since the arousal/valence space is continuous making it a regression problem or at least multiclass and not binary. Authors should re-word this part to make it clear.

Thank you for pointing this out. The reviewer is correct; the original phrase is confusing. Therefore, we have removed this part to avoid any further confusion. Furthermore, this concern has been changed and revised to make it more straightforward and understandable. The revised text was located in the experimental setting subsection under the Method section on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Referring to wearable ECG devices, the authors state: "However, most of these devices store the ECG as images instead of raw numerical data". No references or other market analysis is provided to show that this is the case. The use of ECG wave images for emotion recognition is not properly motivated. I fail to see any advantage of using ECG wave images over the time-series data unless ECG time-series data is not available.

Thank you for drawing our attention to this, and we agree with the comments. Therefore, we have cited the necessary references and revised the statement as per the suggestion. The revised text can be found on page 4 as follows:

“Previous research on human emotions has primarily relied on either direct analysis of 1-D data ^{12 – 14} or the conversion of 1-D data to a 2-D spectral image ¹⁵ prior to identifying the emotions. Despite this, majority of the portable devices record the ECG signal as images (2-D images) in a PDF file rather than as raw numerical data (1-D data) ^{16 – 18}. The example of a PDF-based 2-D ECG is depicted in Figure 2. Due to this problem, researchers were required to convert the PDF file of the ECG into 1-D data before performing further emotion analysis, adding complexity to the pre-processing process. On this account, given the positive results obtained in monitoring and diagnosing cardiovascular-related diseases, the efficacy of 2-D ECG in emotion studies also warrants further investigation.”

How was the data annotated in the DREAMER dataset? Were they continuous annotations or a single annotation per video clip? This is very relevant to include in the article. A few more words about the dataset are needed like a short description of the experimental protocol.

Thank you for bringing this issue to our attention. The short description of the DREAMER dataset has been improved to address this issue. The revised text was located in the Method section under a subsection called "The dataset (DREAMER") on page 8:

“This study used ECG signals from Katsigiannis and Ramzan ¹³ called DREAMER. The DREAMER dataset is a freely accessible database of electroencephalogram (EEG) and electrocardiogram (ECG) signals used in emotion research. However, EEG signals were removed from this study because the primary focus is on ECG signals. The ECG was recorded using the SHIMMER ECG sensor at 256 Hz and stored in 1-D format. The DREAMER dataset contains 414 ECG recordings from 23 subjects who were exposed to 18 audio-visual stimuli designed to evoke emotion. Each participant assessed their emotions on a scale of 1 to 5 for arousal, valence, and dominance. However, because this study was primarily concerned with arousal and valence ratings, participants' evaluations of dominance were discarded.”

Arousal/valence rating values are ranging from 1 to 5 in the DREAMER dataset. The authors never explain how they split them into two classes (high/low) for the binary classification.

Thank you for bringing this to our attention. We have improved this part by adding an explanation of how we categorise emotions into two classes. The explanation can be found on page 13:

“The scale of self-assessed emotions, which ranges from 1 (lowest) to 5 (highest), was classified using a five-point scale with middle-point thresholds (an average rate of 3.8). As a result, scales four and five were assigned to the high class, while the remaining scales were assigned to the low class.”

Authors do not include sufficient information on how the time-series ECG data was converted to an image (resolution, compression, windowing), or how the data was treated in general. Did the authors use the entire ECG signal for each video? Was there any windowing? Since the authors did not properly summarize the dataset (how were the videos annotated?) it is difficult to grasp or guess on how the data was processed.

We agree with this comment. Therefore, we improved the explanation part of pre-processing the 2-D ECG. The improvised text can be found in the first and second paragraphs of subsection 2-D ECG on page 11.

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG (Figure 4). The image has a width of 1920 pixels and a height of 620 pixels.

Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford. The coloured images were converted into greyscale images. Then, binarization of the image using an Otsu's automatic image thresholding method ³⁵ was done. This method ascertains the optimal threshold values from pixel values of 0 to 255 by calculating and evaluating their within-class variance ³⁶”

None of the cited literature used any of the image feature extraction methods that the authors used, and the authors did not discuss their reasoning for why they selected those image feature extraction methods and not the ones established in the ECG image analysis literature that they cited. Some more illustrations of these features may be useful besides the one in Figure 5 in order to convince the readers.

Thank you for pointing this out. Cite literature either using their own algorithm to detect peaks on PQRST waves or automatically extracting ECG features using a deep learning system. However, we had included our reason for employing these image feature extraction methods, which we believed could add a valuable contribution to the state-of-the-art. Additionally, we removed Figure 5 and replaced it with the description of each feature extractor to help convince the readers. The revised text can be found on pages 11 and 12:

“The area of interest for 2-D ECG is laying on the PQRST waves, making the peaks detector the best approach to be employed. Therefore, six different feature extractors that could extract peaks, edges, or corners were applied to extract features from 2-D ECGs using Python version 3.8.5:
1. ORB ³⁷: ORB features are invariant to rotation and noise because they are a combination of Features from Accelerated Segment Test (FAST) detection and Binary Robust Independent Elementary Features (BRIEF) description methods.
2. SIFT ³⁸: SIFT identifies feature points by searching for local maxima on the images using Difference-of-Gaussians (DoG) operators. The description approach generates a 16x16 neighbourhood around each identified feature and sub-blocks the region. SIFT is also rotation and scale invariant.
3. KAZE ³⁹: KAZE is based on the scale of the normalised determinant of the Hessian Matrix, with the maxima of detector responses being captured as feature points using a moving window. Additionally, KAZE makes use of non-linear space via non-linear diffusion filtering to reduce noise while keeping the borders of regions in images.
4. AKAZE ⁴⁰: AKAZE is a more sophisticated version of KAZE that is based on the Hessian Matrix determinant. Scharr filters were employed to enhance the quality of the invariance rotation, rendering AKAZE features rotation- and scale-invariant.
5. BRISK ⁴¹: While searching for maxima in the scale-space pyramid, BRISK detects corners using the AGAST algorithm and filters them using the FAST Corner Score. Additionally, the BRISK description is based on the recognised characteristic direction of each feature, which is necessary for rotation invariance.
6. HOG ⁴²: HOG is a feature descriptor that is used to compute the gradient value for each pixel. The image shape denoted the edge or gradient structure derived using a high-quality local gradient intensity distribution.”

Support vector machine: It is not clear what preprocessing steps were applied to the data. For example, in the DREAMER dataset baseline they only use the last 60s of data for each film clip.

Thank you for this comment. The pre-processing part can be found in the Methods section under the 1-D ECG and 2-D ECG subsections. The pre-processing for 1-D ECG can be read as follows:

“The AUBT and TEAP feature extractors were included with the Low Pass Filter (LPF), a filter meant to reject all undesirable frequencies in a signal. The LPF was one of the most widely used filters before the computation of statistical features for physiological signals ^{31, 32}. As a result, automated 1-D ECG pre-processing utilizing LPF was performed in this study to reduce muscle and respiratory noise in ECG signals.”

Whereas the pre-processing for 2-D ECG can be read as follows:

“The duration of the ECG recording varies according to the duration of the video (average = 199 seconds). As Katsigiannis and Ramzan proposed, this study analysed the final 60 seconds of each recording to allow time for a dominant emotion to emerge ¹³. Following that, 1-D ECG was pre-processed using a simple MATLAB function by ³⁴ to eliminate baseline wander caused by breathing, electrically charged electrodes, or muscle noise. The signal was then divided into four segments corresponding to 15 seconds each. Then, using MATLAB version 9.7, the 1-D ECG was transformed into a 2-D ECG.”

Data was divided 80:20 and 10-fold cross-validation was used. The authors do not specify exactly how the data was split (see DREAMER dataset paper section V as an example). From reading this part, I can only assume that all data from all participants was used for each fold (general model), something that is diverging from how the data was split in the DREAMER dataset paper (they made models for each individual participant). Therefore, any comparisons to the results of the DREAMER baseline are invalid.

Thank you for pointing this out. We have addressed this issue by adding a new subsection under Method, namely, Experimental Setting. This subsection is located on pages 13 and 14, and can be read as follows:

“The hyperparameters for SVM were tuned using an exhaustive parameter search tool, GridSearchCV, from Scikit-learn that automates the tuning procedure ⁴⁵. This study tuned only the parameters with a high and relative tuning risk and left the remainder at their default values because they are the least sensitive to the hyperparameter tuning process, as suggested by Weerts, Mueller, and Vanschoren ⁴⁶.

The dataset was split into a reasonable proportion of training and testing sets to evaluate the model's performance on new unseen data. This study used a stratified train-test split of 80:20 for the training and testing sets. This strategy guarantees the dataset's exact proportions of samples in each class are preserved.

Additionally, as we had a small dataset size, this study applied KFold Cross-Validation, with the number of folds set to 10, the most commonly used number in prior research to improve ERS performance.”

Additionally, the summary of the experimental setting values has been tabulated in Table 5 on page 14.

Results: If the classes are unbalanced (as the DREAMER dataset paper indicates) accuracy is not valid on its own, include f1 score, and/or Cohen's Kappa..

Thank you for your suggestions. As per your recommendations, we have included the F1-score along with the accuracy to address the unbalanced class distribution issue. This result can be found in Table 6 on page 14.

Discussion: "Ref. 11" has nothing to do with the statement in that paragraph. LDA was actually applied in the DREAMER dataset paper and they reported that there were no significant differences in performance.

Thank you for your great observations. We have revised that part as needed. Regarding the LDA, we agree with you. However, as far as we know, LDA was used for classification purposes in the DREAMER dataset paper but not for dimension reduction.

View more View less

Competing Interests

The authors declare that they have no conflict of interest.

Back to all reports

Reviewer Report

18 Views

21 Feb 2022 | for Version 1

Umesh Chandra Pati, Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela, Odisha, India

18 Views Cite this report Responses(1)

Approved With Reservations

Literature survey is poor. More number of related state-of-the-art works should be cited. There is no citation of the work taking into account both ECG image and ECG numerical data.
It has been mentioned that the converted ECG images have been resized to 60% of the original size to reduce the computational time. What is the reason for choosing 60%?
There is no analysis of the computational complexity of the proposed method.
In Table 4, emotion classification accuracies for both ECG image and ECG numerical data have been provided. Can these accuracy values be accepted in practical applications?
There is no comparison of the proposed method with state-of-the-art methods.
References should be complete in all respect and in a uniform style. A few exceptions are 1, 32, 39, 40.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Computer Vision, Image/Video Processing, IoT, Artificial Intelligence.

Respond to this report

Responses (1)

Author Response

30 May 2022

Sharifah Noor Masidayu Sayed Ismail, Faculty of Information Science & Technology, Multimedia University, Bukit Beruang,, 75450, Malaysia

The paper presents a comparison of the Emotion Recognition System (ERS) performance using ECG image and ECG numerical data. DREAMER dataset containing 23 ECG recordings has been used for experimentation. ECG numerical data has been converted to ECG images for comparison. The proposed method consists of feature extraction, dimension reduction, and emotion classification. Arousal and Valence accuracy have been compared using different feature extractors on ECG image and numerical data. ORB, SIFT, KAZE, AKAZE, BRISK, and HOG have been used to extract features from ECG image data whereas TEAP and AUBT have been used to extract features from ECG numerical data. It has been concluded that ECG numerical data provides better performance of emotion classification in comparison to ECG image data.

The authors have done a good job in addressing a typical issue. However, there are many aspects that need attention. Hence, the following suggestions should be addressed to enhance the quality of the manuscript.

Thank you for taking the time to review our manuscript. We reviewed the comments and made improvements based on the comments provided accordingly.

Literature survey is poor. More number of related state-of-the-art works should be cited. There is no citation of the work taking into account both ECG image and ECG numerical data.

We have revised Section Related Works as per the suggestion by adding more related state-of-the-art work citations. The revised part is on pages 5 until 8. We also include the summary information about the existing works that employed 1-D and 2-D ECG input, which is tabulated in Table 2 on pages 6 through 8. Furthermore, the work that considers both ECG images and ECG numerical data is also included in this section. This work can be found on page 6, which reads as follows:

"Additionally, Mandal et al. ⁵ published a study comparing 1-D and 2-D ECGs for the diagnosis of Ventricular Arrhythmia. They concluded that both ECG inputs are effective at detecting the disease."

It has been mentioned that the converted ECG images have been resized to 60% of the original size to reduce the computational time. What is the reason for choosing 60%?

Thank you for drawing our attention to this. We have improved the explanation of this matter in the manuscript. You can find the text on page 11 that reads as follows:

“Due to the fact that the 2-D ECG was converted to a rectangle shape, it is not easy to resize the photos to the standard input image sizes of 224×224 and 299×299. As a result, the converted 2-D ECG was resized to 60% of its original size using Python version 3.8.5. This scale percentage was chosen after considering the quality of the image, the type of feature extractor used, and the computational cost the system can afford.”

There is no analysis of the computational complexity of the proposed method.

Thank you for pointing out this matter. We have included the analysis of the computational complexity under the result section, tabulated in Table 7, which discussed the time analysis of both inputs used to train and test the ERS model. The analysis was located on page 14:

“For comparison purposes, the computation time for both ECG inputs was recorded and reported in Table 7. The average time required to compute 1-D is 1.58 ± 0.07 seconds. In comparison, the average computation time for 2-D is 3377.425 ± 3138.875 seconds. Therefore, according to the observation, 2-D took the longest computation time, whereas 1-D obtained the shortest.”

Additionally, the analysis of the computational time was touched on in the Discussion and Conclusion section as follows:

“In terms of computational cost, 1-D ECG is better to 2-D ECG since it requires less computation time.”

In Table 4, emotion classification accuracies for both ECG image and ECG numerical data have been provided. Can these accuracy values be accepted in practical applications?

Thank you for pointing this out. For your information, our results have been updated according to the result of the latest experiment. This result is tabulated in Table 6 on page 13. Based on this result, the accuracy and F1-score achieved for this study are on par with the existing works, which shows that our ERS model can be accepted in practical applications. However, this does not rule out the possibility of improving this result, as there is much more room to improve the ERS performance to develop a more robust ERS in the future.

There is no comparison of the proposed method with state-of-the-art methods.

A comparison with existing work (DREAMER) has been provided for 1-D ECG. This comparison can be found in the result section on page 15, tabulated in Table 6. However, no existing work using the 2-D ECG of the DREAMER dataset has been reported. Therefore, no comparison can be made.

References should be complete in all respect and in a uniform style. A few exceptions are 1, 32, 39, 40.

Thank you for bringing our attention to this. We have revised the references and citation part accordingly and completed it in all respects and in a uniform style as per suggestions.

View more View less

Competing Interests

The authors declare that they have no conflict of interest.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Mena LJ, et al.: Mobile Personal Health Monitoring for Automated Classification of Electrocardiogram Signals in Elderly. Comput. Math. Methods Med. 2018; 2018(Figure 1): 1–9.

[2] 2. Wang YH, Chung CG, Lin CC, et al.: The Study of the Electrocardiography Monitoring for the Elderly Based on Smart Clothes. 8th Int. Conf. Inf. Sci. Technol. 2018; pp. 478–482.

[3] 3. Ullah A, Anwar SM, Bilal M, et al.: Classification of arrhythmia by using deep learning with 2-D ECG spectral image representation. Remote Sens. 2020; 12(10): 1–14. Publisher Full Text .

[4] 4. Tayel MB, El-Bouridy ME: ECG images classification using artificial neural network based on several feature extraction methods. 2008 Int. Conf. Comput. Eng. Syst. ICCES 2008. 2008; pp. 113–115.

[5] 5. Mohamed B, Issam A, Mohamed A, et al.: ECG Image Classification in Real time based on the Haar-like Features and Artificial Neural Networks. Procedia Comput. Sci. 2015; 73(Awict): 32–39. Publisher Full Text

[6] 6. Wang F, Syeda-Mahmood T, Beymer D: Finding disease similarity by combining ECG with heart auscultation sound. Comput. Cardiol. 2007; 34: 261–264.

[7] 7. Soleymani M, Lichtenauer J, Pun T, et al.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 2012; 3(1): 42–55. Publisher Full Text

[8] 8. Abadi MK, Subramanian R, Kia SM, et al.: DECAF: MEG-Based Multimodal Database for Decoding Affective Physiological Responses. IEEE Trans. Affect. Comput. 2015; 6(3): 209–222. Publisher Full Text

[9] 9. Subramanian R, Wache J, Abadi MK, et al.: ASCERTAIN: Emotion and personality recognition using commercial sensors. IEEE Trans. Affect. Comput. 2018; 9(2): 147–160. Publisher Full Text

[10] 10. Siddharth S, Jung T-P, Sejnowski TJ: Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing. IEEE Trans. Affect. Comput. 2019; 1–1. Publisher Full Text

[11] 11. Ekman P: Cognition & Emotion An argument for basic emotions An Argument for Basic Emotions. COGNlTfON Emot. 1992; 6(May 2013): 169–200. Publisher Full Text

[12] 12. Russell JA: A circumplex model of affect. J. Pers. Soc. Psychol. 1980; 39(6): 1161–1178. Publisher Full Text

[13] 13. Katsigiannis S, Ramzan N: DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals from Wireless Low-cost Off-the-Shelf Devices. IEEE J. Biomed. Heal. Informatics. . 2018; 22(1): 98–107. PubMed Abstract | Publisher Full Text

[14] 14. Yu LC, et al.: Building Chinese affective resources in valence-arousal dimensions. 2016 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. NAACL HLT 2016 - Proc. Conf. 2016; (no. June): pp. 540–545.

[15] 15. Bota PJ, Wang C, Fred ALN, et al.: A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals. IEEE Access. 2019; 7: 140990–141020. Publisher Full Text

[16] 16. Wen WH, Qiu YH, Liu GY: Electrocardiography recording, feature extraction and classification for emotion recognition. 2009 WRI World Congr. Comput. Sci. Inf. Eng. CSIE 2009. 2009; vol. 4: pp. 168–172.

[17] 17. C. U. First Faculty of Medicine: Electrocardiogram - WikiLectures.2018. [Accessed: 04-Oct-2021]. Reference Source

[18] 18. Wagner J: Augsburg biosignal toolbox (aubt). Univ. Augsbg; 2014.

[19] 19. Soleymani M, Villaro-Dixon F, Pun T, et al.: Toolbox for Emotional feAture extraction from Physiological signals (TEAP). Front. ICT. 2017; 4(FEB): 1–7. Publisher Full Text

[20] 20. Minhad KN, Ali SHM, Reaz MBI: Happy-anger emotions classifications from electrocardiogram signal for automobile driving safety and awareness. J. Transp. Heal. 2017; 7(November): 75–89.

[21] 21. Tivatansakul S, Ohkura M: Emotion Recognition using ECG Signals with Local Pattern Description Methods. Int. J. Affect. Eng. 2015; 15(2): 51–61. Publisher Full Text

[22] 22. Song T, Zheng W, Lu C, et al.: MPED: A multi-modal physiological emotion database for discrete emotion recognition. IEEE Access. 2019; 7(October): 12177–12191. Publisher Full Text

[23] 23. Hao P, Gao X, Li Z, et al.: Multi-branch fusion network for Myocardial infarction screening from 12-lead ECG images. Comput. Methods Programs Biomed. 2020; 184: 105286.

[24] 24. Mandal S, Mondal P, Roy AH: Detection of Ventricular Arrhythmia by using Heart rate variability signal and ECG beat image. Biomed. Signal Process. Control. 2021; 68(May): 102692. Publisher Full Text

[25] 25. Otsu N: A Threshold Selection Method from Gray-Level Histograms. IEEE Trans. Syst. Man Cybern. 1979; 9(1): 62–66. Publisher Full Text

[26] 26. Trier OD, Taxt T: Evaluation of Binarization Methods for Document Images. IEEE Trans. Pattern Anal. Mach. Intell. 1995; 17(3): 312–315. Publisher Full Text

[27] 27. Rublee E, Rabaud V, Konolige K, et al.: ORB: An efficient alternative to SIFT or SURF. Proc. IEEE Int. Conf. Comput. Vis. 2011; (May): 2564–2571.

[28] 28. Shi Y, Lv Z, Bi N, et al.: An improved SIFT algorithm for robust emotion recognition under various face poses and illuminations. Neural Comput. Appl. 2020; 32(13): 9267–9281. Publisher Full Text

[29] 29. Alcantarilla PF, Bartoli A, Davison AJ: KAZE features. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 2012; vol. 7577 LNCS(no. PART 6): pp. 214–227.

[30] 30. Tareen SAK, Saleem Z: A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK. 2018 Int. Conf. Comput. Math. Eng. Technol. Inven. Innov. Integr. Socioecon. Dev. iCoMET 2018 - Proc. 2018; vol. 2018-Janua: pp. 1–10.

[31] 31. Liu Y, Zhang H, Guo H, et al.: A FAST-BRISK feature detector with depth information. Sensors (Switzerland). 2018; 18(11). Publisher Full Text

[32] 32. Rathikarani V, Dhanalakshmi P, Vijayakumar K: Automatic ECG Image Classification Using HOG and RPC Features by Template Matching.2016; pp. 117–125.

[33] 33. Ma Y, Wang Z, Wu C: Feature extraction from noisy image using PCNN. Proc. IEEE ICIA 2006-2006 IEEE Int. Conf. Inf. Acquis. 2006; (no. September): pp. 808–813.

[34] 34. Valenzi S, Islam T, Jurica P, et al.: Individual Classification of Emotions Using EEG. J. Biomed. Sci. Eng. 2014; 07(08): 604–620. Publisher Full Text

[35] 35. Velliangiri S, Alagumuthukrishnan S, Thankumar Joseph SI: A Review of Dimensionality Reduction Techniques for Efficient Computation. Procedia Comput. Sci. 2019; 165: 104–111. Publisher Full Text

[36] 36. Bulagang AF, Weng NG, Mountstephens J, et al.: A review of recent approaches for emotion classification using electrocardiography and electrodermography signals. Informatics Med. Unlocked. 2020; 20: 100363. Publisher Full Text

[37] 37. Zhai J, Barreto A: Stress detection in computer users based on digital signal processing of noninvasive physiological variables. Annu. Int. Conf. IEEE Eng. Med. Biol. - Proc. 2006; (no. May): pp. 1355–1358.

[38] 38. Pedregosa F, Varoquaux G, Gramfort A, et al.: Scikit-learn: Machine Learning in Python. J. Machine Learn. Res. 2011; 12: 2825–2830.

[39] 39. Haq S, Jackson P: Multimodal emotion recognition. In: Machine audition: principles, algorithms and systems. Multimodal Emot. Recognit. 2010.

[40] 40. Mitsa T: How Do You Know You Have Enough Training Data?. Towards Data Science. 2019. [Accessed: 07-Jun-2021]. Reference Source

[41] 41. nr-isml: nr-isml/ECG-Numerical-Vs.-Image-Data-for-Emotion-Recognition-System: First release (ECG). Zenodo. 2021.Publisher Full Text

Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system

Abstract

Keywords

Introduction

Emotion model

Figure 1. The valence arousal scale.14

Electrocardiogram & Emotion

Figure 2. The P wave, QRS complex, and T wave in the standard electrocardiogram (ECG).17

Related works

Methods

Proposed method

Figure 3. Structure of emotion recognition system: (a) numerical data and (b) image data.

The dataset (DREAMER)

Table 1. The summary of the dataset.

Experimental setup

Figure 4. The electrocardiogram (ECG) wave image converted from ECG numerical data.

Figure 5. Oriented FAST and rotated BRIEF (ORB) is unable to detect the electrocardiogram (ECG) peaks due to the presence of noise.

ECG numerical data

Table 2. Features extracted from Augsburg Bio-signal Toolbox (AUBT).

Table 3. Features extracted from Toolbox for Emotional feature extraction from Physiological signals (TEAP).

Support vector machine

Results

Table 4. Testing emotion classification accuracy for electrocardiogram (ECG) numerical data and ECG image.

Discussion & conclusions

Data availability

Source data

Extended data

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated

Figure 1. The valence arousal scale.¹⁴

Figure 2. The P wave, QRS complex, and T wave in the standard electrocardiogram (ECG).¹⁷