Automated Emotion Recognition (AER) systems, typically implemented as CAD systems, are important because emotional states can significantly influence human actions and decisions. Automated ER has endless applications in HCI, healthcare, applied learning, driving assistance, marketing, and education. ¹⁸ AER can perform an objective and consistent analysis that was difficult through manual observation alone. ¹⁹ Various Machine Learning (ML) and Deep Learning (DL) methods are employed to process the data from multiple sources to understand the emotion. Hybrid methods are also used for better performance, ¹⁹ as illustrated in Figure 2.

Researchers have conducted numerous studies to distinguish emotions using different modalities to aid the diagnosis, monitoring, and treatment plans. The modalities include non-physiological signals such as speech signal, ^{20, 21} facial expression, ^{22, 23} and text data ^{24, 25} along with physiological signals, such as Galvanic Skin Response (GSR), ^{26, 27} Electrocardiogram (ECG), ^{28, 29} EEG, ^{30, 31} and eye movement signals, ^{32, 33} based emotion recognition also gained popularity as it directly indicates an individual’s emotional state. Since physiological signals have a direct impact from the emotional stimulation, a method for emotion recognition using the recorded physiological signals has a superiority over non-physiological data. It is effective for people who cannot speak or express their feelings externally. The individual cannot control the signal produced in the body by emotional stimulation. ³⁴ EEG is an effective way to capture the emotional state of a person because it provides a real-time insight into the voltage fluctuations in the brain caused by stimulation. ³⁵ Various frequency bands in the brainwave signal ( α, β, γ, θ, and δ) correspond to different emotional states like anger, fear, happiness, sadness, and surprise. ³⁵ Since EEG signals directly reflect brain dynamics, they are considered more reliable than facial expression or speech for emotional recognition. ³⁶ Numerous studies follow a multimodal approach to emotion classification by combining two or more physiological or non-physiological modalities. Deep learning facilitates smooth information fusion across modalities in such systems. EEG combined with any other modality, such as face, ³⁷ eye movement, ³³ speech, ³⁸ and any other physiological signal, ^{39, 40} has been shown to obtain better classification results.

1.1.1 Various deep learning models used for emotion recognition

Traditional machine learning techniques such as k-Nearest Neighbors (KNN), ⁴¹ Random Forest (RF), ⁴² Logistic Regression (LR), ⁴³ Support Vector Machine (SVM), ⁴⁴ and Decision Trees (DT) ⁴⁵ were initially used for emotion classification, which require careful manual feature extraction and feature selection. ⁴⁶ Since it demands significant domain expertise, it could produce biased and subjective results. ⁴⁶ Recently, deep learning techniques - especially different variants of Convolution Neural Networks (CNNs), ⁴⁷ Recurrent Neural Networks (RNNs), ⁴⁸ and hybrid models ⁴⁹ have become popular since they are capable of learning complex patterns and extracting the most relevant features from the raw data, resulting in significant improvements in the performance of the AER system. Each modality has its own advantages and limitations, but physiological signals generally provide a more reliable way to determine emotional states. However, these signals are challenging to process and are highly prone to external noise. On the other hand, speech- and facial expression–based methods are less complex, but since they can be intentionally hidden or manipulated by the subject, accurate analysis becomes difficult. ¹⁸ Multimodal approaches combining data from different sources (e.g., EEG-face, audio-video) are often recommended to enhance the performance. ⁵⁰

EEG-based ER has a wide set of advantages by offering objective and real-time insight into emotional states, which is a positive aspect in terms of healthcare, mental health, and human-computer interaction. ⁵¹ EEG registers brain activity directly, making it trustworthy in contexts that rely on authenticity. Because of the complexity of analyzing the signal manually, machine learning models fail to perform very well on the manually extracted features. Recent developments in the EEG domain have focused on using deep learning techniques and multi-modal integration to increase accuracy and practicality. Attention mechanism and Transfer Learning (TL) techniques have significantly helped in the advancement of EEG-based emotion recognition by improving the feature extraction and model generalization processes. The attention mechanism allows the model to emphasize the most informative EEG channels, frequency bands, and brain regions. ^{52– 54} This extraction of key features is essential for distinguishing emotions. TL techniques allow a model trained on one dataset to adapt to new subjects or new datasets, addressing the individual differences in EEG signals. ⁵⁵

Recently, several research studies have been performed in the field of emotion recognition. Liu et al. ⁵⁶ conducted a study on EEG-based Multimodal Emotion Recognition (EMER), integrating EEG with other biosignals such as Electromyogram (EMG) and ECG for emotion classification. The review by Gkintoni et al. ⁵⁷ discusses DL techniques, including CNNs and RNNs, with a focus on feature extraction methods and their relevance to real-world applications. Erat et al. ⁵⁸ presents deep learning methods as part of classification strategies within the Brain-Computer Interface (BCI) pipeline for EEG-based ER systems. Wang et al. ³⁴ offers a comprehensive review of deep learning models such as CNNs, Deep Belief Networks (DBN), and RNNs, emphasizing their roles in emotion recognition. In a study, the authors categorize DL approaches into CNNs, RNNs, and hybrid models, and also discuss different evaluation strategies such as subject-dependent and subject-independent testing. ⁵⁹ Several other studies highlight the importance of multimodal integration, the utilization of deep learning for automatic feature extraction, and the design of complete EEG emotion recognition pipelines. ^{60, 61} Geng et al. ⁶² reviewed the DL methods based on the feature learning method used in the studies into single, attention-based, domain adaptation-based, and hybrid DL models.

There are a number of clear limitations, even though the reviewed papers offer helpful information about EEG-based emotion recognition. Several papers offer broad overviews without clearly categorizing models based on DL paradigms (supervised, unsupervised, or hybrid), which limits methodological clarity. Some reviews treat DL as part of a larger AI discussion, leading to a lack of focus on DL-specific end-to-end pipelines or evaluation strategies. Only a few studies provide a structured classification of DL approaches and discuss subject-independent evaluations. Moreover, eXplainable AI (XAI), ⁶³ or any other type of model interpretability methods, needs to be addressed, which are critical for building transparent and trustworthy emotion recognition systems. Furthermore, real-time applicability, computational cost, and ethical concerns should also be given importance since they are more important at the practical implementation stage. Figure 3 shows a comparison of our study with the existing reviews on EEG-based ER.

Studies on emotion recognition based on EEG are essential because they offer a direct understanding of the electrical activity of the brain during emotion elicitation, providing a unique, objective, and temporally accurate insight into brain dynamics that no other mode of emotion recognition can offer. ⁵¹ Although multimodal approaches are used to improve accuracy, focusing on EEG alone helps to understand the neural mechanisms of emotion, develop interpretable models, and address the challenges specific to brain signal processing. ⁶⁴ Noise and inter-subject variability are two primary challenges encountered in studies based on EEG signals. EEG signals are less prone to intentional masking or manipulation of emotion compared to facial and voice modalities. Moreover, EEG-based studies have driven innovations in DL, feature extraction, and real-time tracking of emotions, contributing to the foundation of multimodal emotion recognition. Thus, the main objectives of the proposed study are as follows: •

Perform a systematic review of the recent studies on EEG-based emotion recognition, highlighting the methodologies, datasets, and evaluation strategies.

This paper performs a systematic literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, ensuring transparency. ^{65, 66} The review includes peer-reviewed articles selected from five large scientific databases and screened according to the inclusion and exclusion criteria. The main advantages of the proposed study are as follows: •

The study identifies and analyzes how the recent advancements in DL techniques have been applied in the emotion recognition domain.

The paper is organized into the following sections: Section 2 outlines the search strategy followed for identifying and selecting relevant studies. Section 3 presents an analysis of various EEG datasets and the preprocessing techniques commonly used in emotion recognition research. Section 4 discusses DL architectures employed for EEG-based ER systems, highlighting their methodologies, performance, and limitations. Section 5 presents the key challenges and future directions in this domain. Finally, Section 6 concludes the paper with a summary of key insights, followed by standard sections on author contributions, funding information, conflict of interest, data availability, and references.

A search strategy was formulated using specific keywords. The co-occurrence of keywords in the selected studies is represented as network clusters using the VOSviewer tool in Figure 4. ⁶⁸ The terms were grouped into three core categories, including: •

EEG-related terms: “EEG”, “electroencephalogram”, “electroencephalography”

Boolean operators were used (e.g., (“EEG” OR “electroencephalography”) AND (”emotion recognition” OR “emotion identification”) AND (“deep learning” OR “neural network”)) to formulate the queries. The results were restricted to journal articles published between 2020 and 2025, omitting review papers in databases where this option was available.

The initial search yielded a total of 3260 records across all databases. The records were imported to Rayyan AI for the ease of screening and management. Rayyan’s automated deduplication tools identified and removed 886 records, resulting in 2374 unique records. Title and abstract screening were carried out by one reviewer based on the inclusion criteria. Full-text screening was performed on the articles that passed the screening and excluded the articles that did not meet the criteria. A PRISMA 2020 flow diagram, which demonstrates the article selection process is shown in Figure 5.

The following criteria were applied to select studies for inclusion in this review: •

The study must use DL techniques to classify or recognize emotions.

The following studies were excluded from the review: •

Emotion recognition using on ML approaches without the use of DL.

Due to the broad nature of the search query, many irrelevant records were retrieved from the databases, many of which were not EEG-based studies. Rayyan AI’s automated resolution tool removed 886 duplicates, and 2374 unique articles were initially screened. Title and abstract were used for the initial screening process. Many articles were found ineligible, 1712 to be exact, because they did not align with the primary objective of the study. There were 219 articles marked as “maybe” in the Rayyan screening tool; however, 25 out of these were conference papers and were removed. 183 papers were excluded because the papers were later found to be not using EEG as the primary modality. After this, 454 articles were initially shortlisted and 152 articles were removed for not being Q1-ranked after additional eligibility criteria were applied. After full-text screening for significance, 38 more articles were removed from consideration, and 31 papers were excluded for using a multimodal approach. This resulted in a final list of 233 articles to be included in this review.

EEG-enabled emotion detection has gained popularity in recent years, as EEG provides insight into the brain activities induced by certain stimuli, measured using scalp electrodes. ³⁵ It captures the voltage fluctuations in the brain caused by the neuron interactions, providing real-time insight into the emotional state. Various emotional states, such as happiness, sadness, anger, fear, and surprise, correspond to different frequency bands in brainwave signals, including α, β, γ, θ, and δ. Since EEG signals directly reflect brain dynamics, they are considered more reliable than facial expression or speech for emotional recognition. ³⁶ Unfortunately, EEG signals are often noisy and high-dimensional with individual differences across people, which makes the manual analysis challenging and limits the traditional feature engineering methods. ^{97, 98} Thus, an advanced computational method is needed to extract useful information from raw EEG signals, using which an accurate emotion classification can be done.

Deep Neural Networks (DNNs) are advanced computational models inspired by the structure and functional similarities of the human brain. In contrast to traditional algorithms, DNNs are trained on large datasets, enabling them to make accurate predictions by learning more complex patterns in the data. ⁹⁹ A DNN consists of layers of interconnected nodes known as neurons: an input layer to capture data, multiple hidden layers to process and transform data, and an output layer to generate final output. The ability of neurons to receive, process, and forward information facilitates the learning of complex patterns in the data. ¹⁰⁰ The input layer captures unprocessed data such as images, text, and audio. The input is transformed by several neurons in the hidden layers using computations that identify various patterns. Parameters such as weights and biases are optimized during backpropagation by iteratively updating the network to reduce prediction errors. The final output, derived from these internal computations, is produced through the node in the output layer.

Different DL strategies are followed for classifying emotion using EEG signal, as shown in Figure 8. Supervised methods work well with datasets with reliable labelling. The unsupervised method is suitable for identifying hidden patterns in an unlabeled dataset. The hybrid method combines unsupervised and supervised methods, or two supervised methods, for better results. This flexibility of DNN makes it suitable for emotion recognition using the EEG signal.

EEG emotion recognition has evolved over the years with DL. Zheng et al. ¹⁰¹ presented a Deep Belief Network (DBN) trained on the DE features of multichannel EEG to classify positive versus negative emotional states. The DBN achieved accuracies of 86.91%, and a combination of the DBN with the Hidden Markov Model (HMM) provided an accuracy of 87.62%, outperforming the traditional ML methods such as SVM and KNN, which require manual feature engineering. Jirayucharoensak et al. ³¹ introduced a DNN structure that integrates PCA-based covariate shift correction for improved performance in emotion analysis using EEG signal. Using a stacked autoencoder deep network with the PSD as input, PCA chooses the best components. Liu et al. ¹⁰² explored the application of CNNs by converting EEG data into image-like representations after feature extraction. Subsequent research on hybrid deep learning models for EEG emotion recognition frequently references their work, which showed the potential of CNNs in this area.

In supervised DNNs, the model is trained using input signals paired with their corresponding emotion labels (e.g., happy, sad, etc.), and during training, its internal parameters are iteratively adjusted to minimize the difference between actual and predicted emotional states. This is usually done by changing the network’s parameters via the backpropagation algorithm. Supervised DNNs can be further categorized based on the specific task they are designed to perform and the type of data they process.

Multi-layer Perceptron (MLP) is a basic feedforward DNN model that uses non-linear activation functions and backpropagation. ¹⁰³ In EEG emotion recognition, MLPs are primarily used to analyse the emotional state using the handcrafted DE and PSD features. Li et al. ¹⁰⁴ used a Hierarchical 3D MLP-based Neural Network (HMNN) for cross-subject emotion recognition. HMNN uses 3D-MLP Blocks for multi-period EEG feature extraction and fusion. The study highlights individual differences in brain activity that produce high variance in accuracy between subjects. Unlike CNN and RNN, MLPs are not capable of capturing spatial and temporal dependencies effectively, yet they serve as a simple and computationally efficient baseline model for EEG emotion classification.

Building on the idea of learning from limited or unlabeled data, self-supervised EEG emotion recognition explores the unsupervised nature of EEG signals by creating pretext tasks that models need to solve using pseudo labels derived from the data itself. This facilitates the extraction of robust and transferable representations. For example, models such as EEGFuseNet use a hybrid CNN-RNN-GAN network to capture deep spatio-temporal features for EEG, learning with both reconstruction and adversarial features without labels. ¹⁰⁵ The clustering of emotion states is done afterwards using unsupervised clustering (e.g., hypergraph partitioning). Other works, such as Generative Adversarial Network-based Self-supervised data augmentation (GANSER) , employ self-supervision in the form of adversarial training and masking-based tasks, pre-training the network over a large amount of unlabeled data before the supervised classification. ¹⁰⁶ These methods significantly reduce the need for manual annotations and improve the model’s capabilities of generalization with respect to unseen subjects.

Semi-supervised approaches for EEG-based emotion recognition leverage limited labeled data and large amounts of unlabeled examples to achieve efficient learning and potential transferability. Methods such as Semi-supervised and Domain Adversarial learning with EEG (SEDA-EEG) ¹⁰⁷ are based on two-stage training: first, a supervised method is followed to train the model using labeled source domain data, and then it is fine-tuned using domain adversarial learning and pseudo-labeling for the unlabeled target domain data in the absence of the true labels. The pseudo labels obtained from the feature representations of the target are then used in adaptation by combining supervised and unsupervised signals. Another example is Semi-Supervised Domain Adaptation (SSDA) framework, which uses just a few labeled samples per class of each new subject combined with many unlabeled recordings to align feature and prediction distributions. ¹⁰⁸

4.2.1 Convolutional Neural Networks (CNN)

LeCun et al. ⁴⁷ first proposed a backpropagating neural network that learned from 16×16 raw grayscale images without any manual feature engineering. This helped in the development of trainable convolution filters and weight-sharing, which later laid the foundation for the LeNet-5 model. ¹⁰⁹

Initially, the primary purpose of convolutional neural networks is to process grid-like data, such as images or 2D representations. The layers apply convolution operations to extract local and hierarchical features using kernels and filters. For dimensionality reduction, pooling layers are used, followed by Fully Connected (FC) layers for classification, as in Figure 9. CNNs work well for EEG-based ER when the data is converted into a 2D format, such as spectrograms or topographic images, because they can learn meaningful spatial and spectral patterns from the signal. ^{110– 114} However, despite this effectiveness, certain limitations are also observed. These studies highlight the need for a rich EEG dataset and a better channel optimization strategy. They employ data augmentation techniques, and the model’s generalization across subjects is also challenging due to inter-subject variability.

To address these limitations, further improvements were introduced. Large models such as ResNet, ^{115, 116} VGGNet, ⁴⁹ pre-trained on various image datasets, are also used via TL ¹¹⁷ to enhance the feature extraction process from the EEG data. The process of fine-tuning a previously trained model to solve a new problem is known as transfer learning. Primarily, in the medical field, it is costly and difficult to obtain a labeled high-quality dataset. Hence, transfer learning is used to solve this issue. ¹¹⁸ Recent studies explored multimodal fusion by combining features from multiple modalities or decisions of multiple modals for improving classification result. ¹¹⁹ Lian et al. ³⁷ proposed a method for combining EEG and face images, highlighting the need for real-time emotion recognition using multiple modalities. The VGGNet-16 is used as the emotion recognition model for facial images. ¹²⁰ The joint face-EEG model obtains more than 90% accuracy on the valence and arousal dimensions of the MAHNOB-HCI dataset. It has been mentioned that the process is completed in the offline state, and to bring it to a real-world application, it is necessary to use diverse datasets and increase the emotional categories to obtain convincing results. In addition, it suggests the inclusion of the attention mechanism to process the cortical regions of interest.

4.2.2 Recurrent Neural Networks (RNN)

RNNs were proposed in the 1980s and are designed to process sequential data. RNN has a hidden state, unlike feedforward networks, which can capture information from previous steps. ⁴⁸ Figure 10 illustrates the basic structure of an RNN. In this architecture, the input layer feeds into the hidden neurons, and each hidden neuron is linked back to itself through a recurrent connection. The information from the previous step is retained, enabling the processing of sequential data. This makes the RNN suitable for extracting the temporal patterns. By processing sequential EEG data, an RNN can capture the temporal dynamics of the signal, which is crucial for identifying the emotional state.

However, real-time processing of the EEG scalograms is a complex and computationally expensive procedure. ¹²¹ The Attention-based Convolutional RNN (ACRNN) model, introduced by Tao et al. ¹²² uses a CNN to extract spatial information from encoded EEG signals. It also integrates an extended self-attention mechanism on the RNN to capture more discriminative temporal features in subject-dependent experiments, and applies a channel-wise attention mechanism to adaptively weight different channels. A study uses a dual RNN for the temporal feature extraction. ¹²³ However, the vanishing gradient issue with basic RNN networks during backpropagation results in a very small gradient, which stops training or takes a long time, which limits the model’s ability to learn long-term dependencies. ⁵⁹ To overcome this issue, more advanced architectures such as LSTM and GRU are used.

4.2.3 Long Short-Term Memory (LSTM)

In order to solve the issue of vanishing gradients in conventional RNNs, Hochreiter and Schmidhuber ¹²⁴ introduced LSTM. This includes memory cells and gating mechanisms for learning long-range dependencies. LSTM has input, forget, and output gates to regulate the flow and retain information for very long periods, making it suitable for analyzing sequential data such as EEG signals. ⁹² Figure 11 illustrates the internal structure of an LSTM cell. At each step, the forget gate( σ) determines how much of the previous memory to retain. The new memory, c _t is the outcome of the input gate ( σ) and the tanh function. Output gate( σ) controls how much of this new information is exposed, determining the new hidden state h _t.

LSTMs might not be able to fully capture the spatial correlations between EEG channels or may be sensitive to irrelevant features and subject variability. A study used an attention mechanism-based LSTM model to select the appropriate electrodes for emotion classification. ¹²⁵ The results have demonstrated the lobes that correspond to the valence and arousal dimensions. The domain discriminator was designed to learn features that remain consistent across different subjects. However, in subject-independent experiments, variations in data distribution between individuals often reduced performance. To overcome this challenge, researchers combined attention mechanisms with LSTMs, allowing the model to focus on the most emotion-relevant EEG channels and time segments. At the same time, domain discriminators were employed to extract features that generalize better across participants and recording sessions. ⁶²

Oka et al. ¹²⁶ proposed an LSTM-based model enhanced with an attention mechanism and optimized using Particle Swarm Optimization (PSO). While attention helps in improving the extraction of emotion-relevant features and PSO is used to optimize the hyperparameter of the LSTM network. LSTM layers were used for feature extraction in another study, which observed that they can be prone to overfitting when working with smaller datasets. ¹²⁷ But some other research has shown that, with careful application, LSTM networks can significantly enhance the feature extraction process in EEG signals by successfully capturing long-term temporal dependencies that conventional techniques frequently fail to identify. ^{128, 129} As they model both the spatial and temporal features, hybrid models that combine LSTM with CNNs or Graph Convolutional Networks (GCNs) enhance performance by better simulating the brain’s spatial topology and temporal evolution. ¹³⁰ Often, the temporal features of an EEG input extracted by LSTM are fused with the spatial features extracted by CNN to get a spatial-temporal feature representation, resulting in better classification performance. ¹³¹ Yin et al. ³⁰ proposed a fusion model of LSTM and Graph CNN (GCNN), where multiple GCNN modules extract the graph-domain features, and an LSTM layer captures the temporal dynamics from the DE feature cubes. Binary classification on the DEAP dataset achieved better results, suggesting an expansion to multi-class classification. Since these developments have resulted in major improvements in classification robustness and accuracy, LSTM-based architectures are now an essential component of EEG-based emotion recognition research.

4.2.4 Bidirectional Long Short-Term Memory (BiLSTM)

Schuster and Paliwal ¹³² proposed the Bidirectional RNN, which processes data in forward and backward directions using two hidden layers. The BiLSTM is built on this architecture, extending the capabilities of traditional LSTM. Two LSTM networks working in both directions allow access to the past and the future of each time point in the data sequence. The output from both the LSTMs is merged before passing on to the next layer. Emotions that are influenced by subsequent brain activity are captured by the BiLSTM network. Figure 12 shows the working of a BiLSTM network for EEG emotion recognition. At each step t, the input vector x _t (features such as DE, PSD, etc.) is fed into the two LSTMs, one processing the input forward and another in the backward direction. The output y _t is obtained by concatenating the results from both LSTMs, which can then be fed into a neural network or classifier for classification of emotions.

A study adopted BiLSTM to learn spatial-temporal characteristics within and between brain regions, highlighting that conventional LSTM might be insufficient for capturing the full context from EEG features. ¹³³ The development of BiLSTM illustrates the critical importance of enriching contextual understanding in sequential data processing. A study used a hybrid model of BiLSTM and IRNN where the model is trained on the DEAP dataset with 97.19% testing accuracy and tested on the DOSE dataset, achieving 96.29% accuracy. Still, the hybrid model encountered overfitting issues. ¹³⁴ A study performed cross-dataset emotion classification using different models and features. Two benchmark datasets, SEED and DEAP, along with a self-constructed dataset, IDEA, were also considered. The linear-formulation of DE features (LF-DE) along with the BiLSTM model was found to be performing better compared to other models and features for the same set of input. It was one of the few studies to perform cross-dataset emotional classification, and it was found that the accuracy of the classification improved compared to the existing results. But the study comes to the conclusion that the sizes of the datasets used vary and are insufficient, and this restricts the model’s performance. ¹³⁵

Despite these promising results, BiLSTMs alone can be computationally intensive and may not fully utilize the spatial features or handle noise and variability in the EEG data. To address these limitations, researchers have established simpler hybrid models combining BiLSTM with CNNs for spatial feature extraction and resulted in robust and efficient emotion recognition. ¹³⁶ Taking it further, the EWT-3D-CNN-BiLSTM-GRU-AT model applies Empirical Wavelet Transform (EWT) to decompose EEG signals, extracts features, and generates 3D-EEG images capturing spatial, spectral, and temporal information. A 3D CNN learns spatial features, followed by BiLSTM and GRU layers for temporal modeling, and a self-attention mechanism emphasizes emotion-relevant features. The model achieved over 90% accuracy on the DEAP dataset. ¹³⁷

4.2.5 Gated Recurrent Unit (GRU)

GRU was proposed by Cho et al. ¹³⁸ as an effective and simple gating mechanism within RNN. GRU is a simplified version of the LSTM architecture, retaining its ability to capture long-term dependencies. It relies on a reset gate to regulate the data to be forgotten and combines the input and forget gates of an LSTM into a single update gate. GRU is computationally more efficient than LSTM with fewer parameters. In the EEG environment, it strikes a balance between performance and complexity. It learns relevant time-series patterns from the data. This push toward simplicity is very important for lowering the cost of computing, speeding up training, and letting more models be used, especially in places with limited resources. The working of the GRU cell is illustrated in Figure 13. The previous hidden state, h _t−1, and current input x _t are received as input to the cell. The reset gate controls the extent to which information from the previous hidden state is discarded. After this, the input is combined with the reset-modified memory and passed through the tanh activation to generate a candidate hidden state. Finally, the update gate merges this candidate with the previous hidden state to form the new hidden state, h _t.

A study proposed by Cui et al. ¹³⁹ introduces a cross-subject emotion recognition method using GRU and Minimum Class Confusion (GRU-MCC). They use GRU to extract the spatial dependence of the EEG electrodes. The model achieves better accuracy by minimizing the overlap between classes. However, the need for extensive labeled data for adjusting the confusion loss and scalability to unseen datasets is an issue. In a study, GRU is used to calculate the high-level time-domain features for the SSTD model, conducting a subject-independent experiment. ¹⁴⁰ One study combines convolution layers with GRU to capture spatial-spectral features along with temporal features, as GRU does not inherently capture features on the spatial and spectral level. ¹⁴¹ Houssein et al. ¹⁴² used BiGRU to capture temporal features. This study suggested using datasets that better simulate’real-life’ experiences, such as those that utilize video games as an emotion elicitation technique. Table 2 (refer to extended data) summarizes the supervised learning ^{143– 148, 151, 153– 177, 179– 192, 194– 203, 205– 209, 211– 214, 216– 228, 231, 233– 250} studies reported in the past five years.

Unlike supervised methods, unsupervised learning does not rely on labeled data. Objects are identified by groupings within data using methods such as clustering and dimensionality reduction. It also supports generative modeling that can be utilized for data augmentation. ²⁵¹ Deep unsupervised methods include architectures such as Autoencoders (AEs), Generative Adversarial Networks (GANs), Restricted Boltzmann Machines (RBMs), and DBNs. Meaningful representations from complex datasets can be extracted using these methods. RBMs are generative neural networks designed for learning the probability distribution of input data, composed of visible and hidden layers with no intra-layer connections. RBMs are trained using contrastive divergence to capture hidden patterns in the data. DBNs are formed by stacking RBMs, as shown in Figure 14, and fine-tuning the resulting network with backpropagation. ^{252, 253} Hinton et al. ²⁵⁴ proposed the layer-by-layer efficient training, making it feasible to train deep networks with millions of parameters. Later, the model can be fine-tuned on the labeled data. In EEG emotion recognition, DBN can extract features from the raw or preprocessed data without the label. A study used DBN as a baseline to compare with the proposed model, Sparse Dynamic GCNN (DGCNN), which demonstrated superior performance against DBN. ¹⁹³

In the context of EEG-based ER, concepts such as domain adaptation and transfer learning are valuable because they enable models trained on a set of subjects to generalize to new users. ²⁵⁵ This approach allows knowledge learned in a well-annotated source domain to be applied to less-labeled or unlabeled target domains, effectively bridging distributional gaps between individuals. Domain Adversarial Neural Networks (DANN) show promising results in cross-subject and cross-dataset EEG emotion recognition tasks, handling the poor generalization of emotion classification models. ²⁵⁶ The domain discriminator network encourages the system to learn domain-invariant features by focusing on emotion-relevant features. Li et al. ¹¹⁶ proposed a meta-transfer learning strategy using multi-scale features resulting in 71.29% and 71.92% accuracy for valence and arousal on DEAP and 87.05% cross-subject accuracy in SEED datasets. The meta-trained Multi-Scale Residual Network (MSRN) model is fine-tuned on a small number of labeled samples from the target subject for quickly adapting to subject-specific connectivity patterns.

4.3.1 Generative Adversarial Network (GAN)

GAN was proposed by Goodfellow et al. ²⁵⁷ which uses two neural networks that are trained in opposition to one another. GAN is composed of two networks: a generator, which creates synthetic data, and a discriminator, which distinguishes between real and generated data. ²⁵¹ This ‘game’ between these two networks enables the creation of realistic and diverse samples. To overcome the problem of limited labeled data in EEG-based emotion recognition, GAN can be used as a Data Augmentation (DA) tool that favours the real emotional patterns. ²⁵⁸ Figure 15 illustrates the working GAN for the generation of synthetic data for EEG spectrograms. Random noise is fed into the generator network, which creates artificial EEG spectrograms that are then sent to the discriminator. Real EEG samples from the training set were also sent to the discriminator at the same time. It is then backpropagated to the generator to update the parameters after the discriminator classifies it as “real” or “fake” along with the discriminator loss and generator loss. Until the discriminator fails to differentiate between synthetic and real samples, this adversarial training process is continued. ²⁵⁹

To improve the robustness of the proposed Graph Neural Network (GNN), a study utilized GAN-based domain adaptation to augment the dataset. ²⁶⁰ Gu et al. ²⁶¹ proposed a GAN to generate EEG signal representations and combined GCNN and LSTM to identify emotions. However, it has a limitation of modal collapse that most of the GAN-related studies have, where the generator produces similar inputs without capturing the diversity of the data distribution.

4.3.2 Autoencoder (AE)

Rumelhart et al. ²⁶² proposed conventional autoencoders as a type of associative network that learns internal representations by back-propagating errors. It demonstrated the meaningful compressed representation of the input without the support of labels, marking a paradigm shift towards unsupervised learning. ²⁶³ Autoencoder has an encoder and a decoder. The encoder maps the input data into a lower-dimensional latent space, while the decoder reconstructs the original input from this compressed representation by minimizing reconstruction error, as shown in Figure 16. The autoencoder will learn meaningful features from the input data through reconstruction. Variational Autoencoders (VAE), Sparse Autoencoders (SAE), and Denoising Autoencoders (DAE) are some of the varieties. In the EEG environment, they are used for feature extraction and dimensionality reduction. Irrelevant noise in the signal can be filtered out while capturing the most important features. The latent representation can be given as input to a classifier or can be clustered.

Wang et al. ²⁶⁴ uses a Multi-Modal Domain Adaptive VAE (MMDA-VAE) to provide a combined representation of multiple modalities for enhancing cross-domain emotional classification on SEED and SEED-IV. In this study, a VAE is used to project data from various modalities onto a single latent space. While the model achieves improved performance, its limitations are also noted. The encoder and decoder in VAE produce a large number of parameters. The method assigns the same weight to the two modalities. So, it is suggested to use an adaptive weighting mechanism in the upcoming studies.

It is also advised to use additional physiological signals such as GSR, EMG, and ECG. Li et al. ²⁶⁵ uses a VAE for learning the spatial-temporal latent features. The class imbalance problem in the dataset was also highlighted in this study. Liu et al. ¹⁰² used a CNN-SAE-DNN model for emotion classification, where CNN and SAE are used for feature extraction and DNN for the final classification. In order to improve performance, the author recommends that different AE designs, including stacked or variational autoencoders, be investigated in the future. It is also pointed out that the label information was used in the feature extraction, which raises questions about biases and overfitting. Pang et al. ²⁶⁶ proposed the Multi-Scale Masked Autoencoders (MSMAE) model, which is trained on a large scale of unlabeled EEG data to extract subject-invariant features. These features are then fine-tuned on a small amount of labeled data from a specific subject for personalization. An attention feature extractor, Attn, is used for aligning features of pre-training and fine-tuning data. But the current performance of the MSMAE model relies on the handcrafted features, resulting in information loss. An overview of recent unsupervised methods ^{267– 288} from the last five years is presented in Table 3 (refer to extended data).

Hybrid learning is a combination of supervised and unsupervised methods. Supervised models perform better on labeled data but cannot learn from unlabeled data. Generative models are flexible because they can learn from both labeled and unlabeled data. The hybrid method combines both approaches to benefit from their individual strengths. These semi-supervised methods can benefit from small labeled datasets while utilizing a large amount of unlabeled data to find patterns and increase robustness. Hybrid models can be a combination of (1) different supervised or unsupervised methods (e.g., CNN+RNN, GAN+AE) or (2) a supervised and an unsupervised model (e.g., AE+CNN, GAN+GCN).

Integrating CNN and RNNs has become common in EEG-based emotion recognition because CNNs extract spatial and spectral features from EEG input, while RNNs like LSTM or GRU extract temporal dynamics. Zhang et al. ²⁸³ uses a stacked depthwise separable CNN for extracting spatial and spectral features and an LSTM for temporal information, incorporating multiple attention mechanisms. A 4D feature representation is constructed of DE and Absolute Power (AP) features from four frequency bands. However, the feature representation is weaker due to the use of 2D mapping matrices for representing spatial relationships among electrodes. MobileNet RNN (MRNN) proposed in uses a pre-trained MobileNet model together with an RNN for spatial and temporal feature extraction, respectively. ¹²¹ While it captures EEG dependencies effectively, it poses challenges in real-time applications due to computational load. The limited data scale also restricts improvements in deep neural network-based emotion recognition. Li et al. ²⁸⁴ proposed Spiking Spatiotemporal Neural Architecture Search (SSTNAS), combining spiking CNN for spatial and a spiking LSTM for temporal feature extraction. The study used the XAI tool Shapley Additive exPlanations (SHAP) for model interpretability and pointed out the high computational cost and poor cross-dataset performance. The need for lightweight models and domain generalization is emphasized.

Generative models are widely used in hybrid approaches. In a study, an Extreme Learning Machine Wavelet Autoencoder (ELM-W-AE) was used for data augmentation. ²⁸⁵ Even after ResNet-18 was used for classification with 99.6% accuracy, the study fails to explain the model’s generalization ability. Mohajelin et al. ²⁸⁶ and Gilakjani and Osman ²⁶⁰ use GANs for augmenting the dataset and Graph Neural Networks (GNNs) for classification, highlighting the limited emotional categories present in current databases. A study used a VAE-GAN model for generating high-quality artificial samples by segmenting DE features temporally and spatially. ²⁸⁷ However, the network has a complex structure and relatively high time complexity. An unsupervised VAE was used to learn spatio-temporal latent features without labels in a study. ²⁶⁵ A parallel branch with GCN and GRU extracted spatio-spectral features from labeled data. The representations from both branches were fused to obtain subject-independent emotion classification. The model employed a multi-task training strategy, increasing computational demand and complexity due to the dual branches. Zhang et al. ¹⁰⁶ proposed a self-supervised data augmentation framework using a masking reconstruction GAN that reconstructs masked portions of EEG signals. The synthesized EEG samples are then used along with real EEG data to fine-tune a supervised classifier. According to the authors, the current classifier, STNet, is not designed to handle distribution shift across subjects.

Mai et al. ²⁸⁸ proposes a wearable EEG system focused on EEG signals from the ear rather than traditional scalp EEG devices. The system uses a superlets-based signal-to-image conversion framework, which transforms EEG signals into 2D-images, making it convenient to process using a modified ViT integrated with Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA). The model achieved an accuracy of 92.39% on the self-collected dataset, outperforming ResNet-50 and EfficientNet-B0 models. Even though the model offers limited spatial coverage causing a potential data loss. In the DRS-Net model, ²⁸⁹ a dynamic reservoir-state encoder is employed to capture spatio-temporal features from multi-channel EEG data. The extracted features are then processed through an LSTM-dense decoder for emotional state classification. The reservoir computing processes the temporal sequences efficiently, and LSTM handles long-term dependencies. The model performance drops on subject-independent experiments. Zhang et al., ²⁹⁰ proposed a Python toolbox built on PyTorch, named TorchEEG _EMO , where the workflow is divided into five distinct modules (dataset, transforms, model selection, models, and trainers) to provide plug-and-play functionalities. Also, a window-centric input-output system is introduced. The model provides almost all popular EEG datasets, preprocessing techniques, state-of-the-art EEG models, and evaluation strategies. In a study, a Semi-skipping-Layered Gated Unit (SLGU) was used to automatically skip the divergent factor during the network training. ²⁹¹ SLGU-ENet was used for deep feature extraction and Support Vector Networks (SVN), Naive Bayes (NB), and KNN were used for classification. In order to reduce the computational cost a reduction function named bag of visualized characters (BoVC) is used.

Wang et al. ²⁹² proposed a knowledge distillation-based model, where a large transformer-based teacher model was trained on labeled source EEG samples. A lightweight BiLSTM student model mimics the teacher’s feature representation and is further refined using Domain Adversarial Neural Network (DANN) by aligning features between labeled source and unlabeled target domains. The model’s performance depends on the generalization ability of the teacher and it is computationally expensive than single stage method. The DC-ASTGCN model combines a Deep CNN (DCNN) with an Adaptive Spatiotemporal GCN (ASTGCN) enhanced with an attention mechanism and adaptive modules, enabling the extraction of both local frequency-domain features and spatio-temporal connectivity patterns from EEG signals. ²⁹³ Hybrid Model with Improved Feature set for Emotion Detection (HMIFED) employs a hybrid architecture that combines BiLSTM and an improved RNN (IRNN) for cross-dataset emotion recognition. ¹³⁴ Although the model achieves good accuracy, it still encounters overfitting issues and lacks generalization. CIT-EmotionNet consists of a CNN and Transformer network module processed simultaneously. The CNN captures local image features, while the Transformer captures global dependencies. ²⁹⁴ CIT-EmotionNet achieved a maximum average recognition accuracy of 98.57% on the SEED dataset and 92.09% on the SEED-IV dataset. Table 4 (refer to extended data) provides a detailed summary of hybrid approaches ^{295– 335, 337– 343} investigated over the last five years.

Graph Convolution Networks (GCN) are used in emotion recognition because they can process non-Euclidean and graph-like structures of EEG data, capturing the spatial relationships between EEG electrodes very well. Song et al. ²³² proposed the Graph-Embedded CNN (GECNN) model, combining a CNN and a graph module for extracting both local and global features. Both subject-dependent and independent operations were performed on SEED, SDEA, DREAMER, and MPED datasets. There is a gradual drop in the performance of the model for subject-independent evaluation. Since the model focuses on the local and global spatial features, temporal information is rarely considered. An Ordinary Differential Equation (ODE)-based GCN was proposed by Chen Y et al. ¹²⁹ combining ODE-driven spatial propagation and Dynamic Time Wrapping (DTW)-based temporal alignment for improved valence/arousal classification. However, the proposed method assumes smooth and continuous EEG signal with stable emotional transitions according to “homophily” assumption, making it less effective when abrupt signal or emotion fluctuation occur. The hierarchical dynamic GCN proposed in a study uses a gated graph convolution to capture both intra- and inter-region dependencies. ²¹⁰ A subject-independent experiment on the SEED dataset provides 89.23% accuracy. Siam-GCAN introduces a twin-branch GCN with shared weights to process pairs of EEG trials. Both branches apply the same graph convolutions over the electrode adjacency graph, followed by a multihead attention mechanism. ²³⁰ Recently, models based on Capsule Networks (CapsNet) have shown strong performance efficiency in emotion recognition, addressing challenges like cross-subject variability, spatial feature extraction, and channel redundancy. ³⁴⁴ In their work, Wei et al. ³³¹ developed TC-Net, which leverages a Transformer module to extract global features and an Emotion Capsule module to model spatial features among EEG channels, achieving good results (≈98%) on both DEAP and DREAMER datasets. The study focused only on the subject-dependent scenario. The DA-CapsNet model introduced a multi-branch CapsNet, where each branch processes DE features from different frequency bands. ²⁷³ By integrating domain adaptation, the model significantly improves cross-subject emotion recognition across DEAP, DREAMER, and SEED datasets. The model is highly dependent on the quality of domain adaptation. Another model, Bi-CapsNet, employs binary weights and activations to lower computational cost and memory usage while still maintaining strong accuracy, making it suitable for deployment on mobile devices. ²⁰⁴ It achieves 25× reduced computational complexity and 5× less memory usage, while only less than 1% drop in accuracy. The accuracy of the model is highly reduced when directly used in the subject-independent scenario. The Hierarchical Attention-CapsNet(HA-CapsNet) model integrates 3DCNN and CapsNet with a hierarchical attention mechanism (local and global attention), extracting rich spatio-temporal data from multidimensional EEG data. ⁵³ The generalization of the model to more diverse, real-world data is not available. Jana et al. ²²⁹ proposed a method where sparse spatio-temporal frames are created from EEG data, combining the position of electrodes and temporal dynamics. Capsule neural dynamics are utilized to model both local and global spatial-temporal information. The accuracy of the model seems to be reducing during the cross-subject experiment, showing the difficulties in generalizing the model to unseen subjects. Several other studies also use capsule networks as spatial features extractors and classifier achieving better classification results all over. ^{96, 150, 215, 301}

Concepts such as XAI or uncertainty analysis are important for a DL-based model because they make it more interpretable, trustworthy, and relevant. ⁶³ XAI helps to reveal how and why a deep learning model predicted that specific emotion based on the given EEG signal. To build trust in the results of affective computing and BCI systems, interpretability is important. XAI will help in highlighting the regions or frequencies of the brain that contribute to the result, which will improve the clinical explanation of the model. Similarly, uncertainty analysis quantifies the confidence the model has in its predictions. ³⁴⁵

EEG-ConvNet proposed by Khan et al. ¹¹¹ used the XAI techniques Gradient Class Activation Mapping (Grad-CAM) ³⁴⁶ and Integrated Gradients (IG) to interpret the predictions. The bagging ensemble approach is utilized for robustness, and the IG method is employed on the ResNet-34 model. Chaudary et al. ¹⁵² proposed the souping of EEG-CNN models trained on EEG scalograms of different sizes. It also incorporates Grad-CAM visualization for the interpretability of the model. Chen et al. ³⁴⁷ used uncertainty to guide the augmentation of EEG graph data. Uncertainty is quantified and utilized to adaptively augment graph connectivity. It enhances the module’s ability to handle distribution shifts in unseen subjects. In the model Connectivity Uncertainty GCN (CU-GCN), uncertainty is used to represent the spatial dependencies and temporal-spectral relationships in EEG signals, guiding the construction of the adjacency matrix through a Bayesian framework. ¹⁷⁸

This review is mostly focused on EEG-based unimodal emotion recognition, but it’s important to note that multimodal techniques that combine other physiological and behavioural data have made a lot of progress. Multimodal emotion recognition models are often developed by pairing EEG with behavioural or physiological modalities (like facial expression or GSR) to provide a richer understanding of emotion responses. These models can usually perform better than unimodal systems in cross-subject emotion classification tasks because the brain-level signals (EEG) can provide clearer representations of emotional states. ³⁴⁸ Cui et al. ³⁴⁹ proposed a multimodal approach combining EEG and facial expression, using a cross-modal attention that fuses the EEG vector with a ConvLSTM (with spatial-attention) face encoder for classification. The multimodal approach provides better results than using a single modality (face or EEG). Using the MAHNOB-HCI data set, Sedehi et al. ³⁵⁰ examined the causal relationship between pupil dilation levels, EEG and ECG. Granger-causality maps are used to analyze causation, and the ResNet-18 model is used for classification. ³⁵¹ Irrelevant frames are discarded to enhance the signal quality. But the method is limited to only two emotional classes. Pan et al. ³⁵² propose Deep-Emotion, a multi-branch architecture that fuses EEG, facial, and speech inputs using GhostNet, Lightweight Full CNN (LFCNN), and tree-like LSTM (tLSTM), respectively. A decision-level fusion method was used to combine the results from multiple modalities, resulting in a more accurate and collective result. The authors question the generalizability of the model using the public datasets in the real-time scenario. Li et al. ³⁵³ proposed a Deep EEG-first Multi-physiological Affect (DEMA) that combined EEG with other physiological signals such as blood pressure, GSR, respiration belt, and temperature. The model introduced the Affective Influence Matrix (AIM) to align and unify multimodal representation by assessing the influence of EEG on the modalities. Fu et al. ³² proposed a multimodal approach combining EEG with eye movement signals, where a feature guidance module is used to direct the extraction of eye movement features. The objectivity of emotion estimation and individual differences are two of the problems highlighted.

In conclusion, recent multimodal emotion recognition literature has repeatedly demonstrated that emotion classification performance and real-world robustness will be improved through the integration of EEG data with other physiological or behavioural signals (facial expression, speech, ECG, and eye movement). Today, many of the most recent state-of-the-art DL methods that use cross-modal attention, transformers, and domain adaptation have made the process of combining these independent types of data even more effective. The combination of the complementary features of the different modalities by deep learning will help to provide users with advanced emotional recognition systems that offer even less biased and more reliable emotional recognition. But multimodal approaches have their own challenges, including increased system complexity, the need for synchronized data collection, and the selection of an appropriate fusion strategy. Yet multimodal approaches have the capacity to aid the advancements in the field of affective computing, enabling more natural emotion computing systems and human-computer interaction.

•

Challenges in data representation: Several EEG emotion recognition studies have analyzed data from EEG data collected in controlled laboratory settings with passive stimuli, such as videos, images, or sounds. In contrast, emotional states in real-world situations are in fact more often dynamic, complex, and driven by some situations. The difference between EEG collected in a lab setting and emotional states in the natural environment potentially raises concerns with reliability. Models trained on experimental data may have difficulty in generalizing when it is applied to real-world situations such as classrooms, workplaces, or health care settings.

There are a number of promising strategies and research directions that can be used to get around the problems that are currently making EEG-based emotion recognition systems less useful and effective: •

There is a growing demand to go beyond traditional laboratory-based video clips, images, and audio-based emotion elicitation. While controlled experimental designs provide precision, they do not specifically reflect the real-life characteristics of emotion. For the field to evolve, future research should collect EEG data in more natural real-life scenarios, such as games and virtual reality, that more accurately reflect the complexities of emotional experience encountered daily. To analyze the temporal trajectory of the emotions, the emotional history of the person can be considered. This could include continuous data collection for a long period using an emotionally rich stimulus.