Motion and Geometric Feature Analysis for Real-time Automatic Micro-expression Recognition Systems

Introduction

A micro-expression is a brief, spontaneous facial expression that occurs on a human face in response to the emotions they are experiencing. The micro-expression contains a significant amount of information, and has attracted the interest of computer vision researchers because of its potential uses in security, interrogation, and healthcare.^1-3 However, due to the speed of facial muscle movement, it is difficult to extract this information, and the features must be more detailed. A typical period of micro-expression is 200 milliseconds or less.⁴ For real-time micro-expression analysis and emotion recognition, the process involves pre-processing, feature extraction, and recognition. This paper examines two popular feature extraction approaches: motion-based features and geometric-based features. These approaches are reported to have reliable details from uncontrolled image data, which makes it feasible for real-time analysis.

A motion-based feature is constructed based on non- rigid motion changes of subtle expressions where motion changes are extracted for spotting purposes. Facial motion analysis was first presented in⁵ using optical flow to spot micro-expressions. Since then, several studies in this field have explored this approach for facial landmark detection and micro-expression recognition. In,⁶ authors proposed an optical flow features from Apex- frame network (OFF-ApexNet), which combines optical flow guided context with the convolutional neural network (CNN) to compute features. Then, authors in⁷ presented a novel algorithm that combines a deep multi-task convolutional network for detecting facial landmarks and a fused deep convolutional network for micro-expression features. In another study,⁸ authors suggested the Riesz pyramid and a multi-scale steerable Hilbert transform. While, Merghani and Yap⁹ proposed a new region-based method with an adaptive mask. However, the motion-based features reported recognition accuracy from the current studies peaks at 74.06% over the CASMEII using leave-one-subject-out cross validation (LOSOCV).⁶

On the other hand, geometric facial analysis deals with the locations and shapes of facial components. As highlighted in Liu et al.,¹⁰ performance of landmark detection algorithms is limited, where only a few studies utilized landmarks in early facial graph representations. However, with recent advancements in face analysis study, improved facial landmark detection algorithms are presented in several studies.^11-14 Towards facial landmark graph features, Lei et al.,¹⁵ presented a method that only employed 28 brow and lip landmarks, which contributed significantly to micro-expressions. While, other studies^16-19 presented graph-based methods using AU to define landmarks of interest. The reported recognition accuracies from these methods proved that micro-expression features can be extracted using facial graph approaches. However, the general problem with graph-based micro-expression recognition is the lack of large-scale in-the-wild datasets. To date, the recognition accuracy peaks at 87.33% over the SAMM dataset with LOSOCV as reported in Buhari et al.¹⁸

Methods

An automatic micro-expression recognition system is implemented for real-time facial analysis by integrating face landmark detection, feature extraction, and then classification. In the developed system, a trained model is generated using publicly available spontaneous micro-expression datasets. For micro-expression feature analysis, two different methods were implemented. Firstly, a Bi-Weighted Oriented Optical Flow (Bi-WOOF) feature descriptor by Liong et al.,²⁰ is used. This method is a motion-based approach that uses optical flow to compute features, and it requires an apex-frame spotting before the feature computation. The BiWOOF is considered in this study as the results of its performance improvement over the textural feature extraction methods such as local binary patterns on three orthogonal planes (LBP-TOP) as reported in Liong et al.²⁰ However, the computational cost poses challenges for real-time recognition as it require apex-frame spotting. The second feature descriptor considered is a full-face graph by Buhari et al.¹⁸ This geometric-based feature computation method requires only facial landmarks to compute features. The full-face graph is considered in this study because the computational time is significantly low in comparison with motion-based. However, it is reported that the earlier geometric-based methods could not detect hidden changes in facial components due to its subtleness and briefness.

Motion-based framework

Figure 1 illustrates the implemented real-time micro-expression recognition system developed using Bi-WOOF. This feature extractor requires at least two frames (i.e, neutral frame and apex-frame) to compute. Firstly, the system captures images of faces using dlib-19.4.²¹ Next, apex-frame spotting is applied using an automatic apex frame spotting method by Liong et al.,²² to identify the frame with the highest facial expression within the captured image sequences (i.e., processing sample). As reported by the authors in Liong et al.,²² the performance of their method improved over the annotated apex frames provided in micro-expression databases. Here, image sequences from spontaneous micro-expression datasets were utilized. Upon identifying the onset and apex frames, optical flow vectors are computed to define the face motion patterns: (i) magnitude: pixel movement intensity; (ii) orientation: flow motion direction; and (iii) optical strain: modest deformation intensity. Then, using the computed optical flow vectors (i.e., the magnitude, orientation, and the optical strain), Bi-WOOF features are formed. Step-by-step details of this method can be found in Liong et al.²⁰ Figure 1 shows a framework of the real-time micro-expression recognition system using apex-frame spotting and BiWOOF feature extraction method.

Figure 1. Framework of the designed real-time micro-expression recognition system.

Geometric-based framework

Figure 2 illustrates the implemented real-time micro-expression recognition system developed using full-face graph. At first, facial landmark detection is applied to detect coordinates of facial components using dlib-19.4. Then, segments of lines are generated using the detected coordinates of facial components by connecting each landmark point (denoted as $p$) with subsequent landmark points (denoted as $q$), for $p\in \left\{\mathrm{1,2,}\dots \mathrm{,}N\right\}$ and $q\in \left\{\mathrm{1,2,}\dots \mathrm{,}p\right\}$, where $N=68$. This concept is described as a full-facial graph using landmark points, and segments are generated as follows:

Figure 2. Frame-based micro-expression recognition system.

$$\Im (k)=\left\{p\mathrm{,}q\right\}\to k\in \left\{\mathrm{1,2,}\dots \mathrm{,}{\displaystyle \sum }p\times q\right\}\mathrm{.}$$

The indexes (i.e., $p\mathrm{,}q$) of every landmark with subsequent landmark points are determined and stored as segments in ℑ for feature computations. After the graph is generated, features are computed by calculating the Euclidean distance and gradient of every segment, an idea presented in Buhari et al.¹⁸ The total number of features computed using this technique is $N\times (N-1)\to K$, which translates to 4,556 features at $N=68$.

To further analyse the potential performance improvement of the geometric-based features, Eulerian motion magnification (EMM) is applied to the images to amplify the micro-expressions prior to the landmark detection process. Eulerian-inspired approaches^23,24 do not require explicit motion vectors but emulate motion magnification by magnifying property changes, such as amplitude (denoted as A-EMM) or phase (denoted as P-EMM). According to Le et al.²⁴ A-EMM outperformed P-EMM in terms of recognition rates over a broad range of magnification factors. Thus, this paper considered the A-EMM to the images before the feature computations. Details of the methods for A-EMM are detailed in Le et al.²⁴ Figure 3 illustrates the principle of the integrated a magnification sub-process to the implemented single-frame sample with a geometric-based features system.

Figure 3. Frame-based micro-expression recognition system with A-EMM.

Experiment settings

The experiments were performed using four spontaneous datasets: (i) spontaneous micro-expression dataset (SMIC) dataset,²⁵ (ii) Chinese Academy of Sciences Micro-expression (CASMEII) dataset,²⁶ (iii) spontaneous macro-expressions and micro-expressions (CAS (ME)²) dataset,²⁷ and (iv) spontaneous actions and micro-movements (SAMM) dataset.²⁸ These are spontaneous micro-expression datasets which were used in this study with full details of these datasets in Li et al.,²⁵ Yan et al.,²⁶ Qu et al.,²⁷ and Davison et al.²⁸ Details to acquire these datasets used in this study are available at www.oulu.fi/cmvs/node/41319 for SMIC,²⁵ fu.psych.ac.cn/CASME/casme2-en.php for CASMEII,²⁶ fu.psych.ac.cn/CASME/cas (me)2-en.php for CAS (ME)²,²⁷ and personalpages.manchester.ac.uk/staff/adrian.davison/SAMM.html for SAMM.²⁸ Moreover, to evaluate the performance using a larger dataset, this paper merged the four datasets to form a COMBINED dataset. The COMBINED dataset is created from the raw images of all the four datasets. The steps for generating the COMBINED datasets includes; face detection, face cropping, colour-space conversion to grayscale, and image re-scaling to $140\times 170$. From these steps, colour-space conversion is applied to adopt the SAMM dataset samples as provided in grayscale format, where the sample re-scaling to $140\times 170$ adopts the SMIC dataset cropped image sizes (i.e., the smallest cropped image size considered to provide reliable features description, and achieve high speed performance for real-time micro-expression recognition). The image re-scaling utilises a down-sampling technique by Buhari et al.,²⁹ in order to re-produce a high quality down-scaled samples. In addition, the COMBINED dataset adopted the SMIC dataset sample labelling by re-grouping the seven classes of emotions (i.e., happiness, sadness, anger, surprise, fear, contempt, and disgust) to three classes (i.e., positive, negative, and surprise). Here, positive $\in \left\{\text{happiness}\right\}$, negative $\in \left\{\text{sadness,}\text{anger,}\text{fear,}\text{contempt,}\text{disgust}\right\}$, and surprise $\in \left\{\text{surprise}\right\}$. Figure 4 illustrates the COMBINED dataset formation from the four spontaneous datasets. Note that the participant images utilised in Figure 4 are the publishable images with the consent of participants, as stated in the documentation from each study. While, Table 1 summarises the selected spontaneous micro-expression datasets used in this study. In this table, the COMBINED dataset is denoted as $\delta$.

Figure 4. COMBINED dataset formation, sample images were taken from SMIC, CASMEII, CAS (ME)², and SAMM datasets as labelled.

Table 1. Summary of spontaneous micro-expression datasets for analysis.

Datasets	Frame rate	Subjects	Samples	Classes
SMIC	100	20	164	3
CASMEII	200	35	247	5
CAS (ME)2	30	22	341	3
SAMM	200	32	159	7
$\delta$	$-$	94	911	3

Results

Table 2 presents the recognition accuracies of the baseline Bi-WOOF (denoted as BBW), the optimized Bi-WOOF (denoted as OBW), full-face graph (denoted as FFG) and the full-face graph with A-EMM (denoted as FFG+M). Here, the BBW and OBW yield the highest recognition accuracies of 66.01% and 69.15%, respectively, over the COMBINED dataset. Similarly, FFG and FFG+M yield the highest recognition accuracy of 77.05% and 77.85%, respectively, over the COMBINED dataset. From these results, it is observed that the OBW improved the performance of the BBW by up to 3.28% over the SAMM dataset. On the other hand, the implemented full-face graph with A-EMM improved the performance by up to 1.20% over the SAMM dataset. Then, in comparison with optimized Bi-WOOF and full-graph with A-EMM improved the performance by 9.72%, 11.23%, 18.32%, 7.03% and 8.7% over SMIC, CASMEII, CAS (ME)², SAMM and COMBINED datasets, respectively. Moreover, Table 3 compares the accuracies of the optimized BiWOOF with other motion-based methods, where Table 4 compares the accuracies of the full-face graph with other geometric-based methods.

Discussion

Table 3 lists the performance of benchmark motion-based methods,^6-9,20,31,32 with the optimized Bi-WOOF. The best reported accuracy is 74.06% over the CASMEII dataset.⁶ Looking at Bi-WOOF+Phase,³² the highest performance reported is 68.29% over the SMIC dataset, which outperformed the baseline and the optimized Bi-WOOFs. However, the optimized Bi-WOOF outperformed the reported accuracies in other studies,^7-9,31 as shown in Table 3.

On the other hand, Table 4 lists the benchmark geometric-based methods^15-19 with the full-face graph and full-face graph + A-EMM, which are denoted as experiment I and experiment II, respectively. From these results, Buhari et al.,¹⁸ reported the highest accuracies of 76.67%, 75.04%, 81.41%, and 87.33% over the SMIC, CASMEII, CAS (ME)², and SAMM datasets, respectively. In Buhari et al.,¹⁸ the full-face graph utilized 68 landmarks from the raw images (denoted as $ℝ$). While, the full-face graph in experiment I and experiment II yield 74.62%, 74.41%, 75.11%, 74.33% and 75.01%, 74.55%, 76.21%, 75.53% over the SMIC, CASMEII, CAS (ME)² and SAMM datasets, respectively. From these results, the full-face graph in experiment II outperformed the full-face graph presented in¹⁸ with 8.11%, 1.1%, and 3.38% over the SMIC, CASMEII, and CAS (ME)² datasets, respectively. While, on the other hand, Buhari et al.,¹⁸ outperformed the results in experiment II with 4.7% over the SAMM dataset. While, in comparison with reported accuracies presented in Table 3, the full-face graph in experiment II achieved the highest performance.

Looking at the performance presented in Tables 3 and 4, the performance reported in Buhari et al.,¹⁸ registered the highest accuracy of 87.33% over SAMM datasets, respectively. While, the full- face with A-EMM outperformed the full-face graph performance presented in Buhari et al.¹⁸ From these results, the conclusion can be drawn that the geometric-based features are competing closely with the motion-based features. In terms of the computational time, the optimized Bi-WOOF running time is 0.36 seconds per sample (i.e., 2.7fps), while the full-face graph running time is 0.10 seconds per sample (i.e., 10fps), with a 640×480 image resolution, on an Intel i5-3470 machine. The reported running times include facial landmark detection and classification.

Conclusions

This paper analyzed the performance of motion-based features (i.e., Bi-WOOF) and geometric-based features (i.e., Full-face graph) for real-time micro-expression recognition systems. The results indicate that the optimized Bi-WOOF improved recognition accuracy of the baseline Bi-WOOF by up to 3.28% over the SAMM dataset. While, on the other hand, the full-face graph performance is improved by up to 1.20% with A-EMM over the SAMM dataset. Moreover, the full-face graph and full-face graph with A-EMM exhibit significant performance improvement over the baseline and the optimized Bi-WOOF by up to 18.32%. Though the full-face graph improved performance of recognition accuracy, the processing time could limit the readiness of the full-face graph features for real-time systems using high-speed cameras.

Data availability

Author contributions

Conceptualization, Investigation, Methodology, Formal Analysis, Software, Visualization, Writing – Original Draft Preparation, Review & Editing; AM Buhari: Conceptualization, Resources, Formal Analysis, Methodology, Supervision, Writing – Review & Editing; CP Ooi: Conceptualization, Formal Analysis, Supervision, Validation, Writing – Review & Editing; VM Baskaran. Conceptualization, Methodology, Writing – Review & Editing; WH Tan.

Competing interests

No competing interests were declared.

Grant information

The authors declared that no grants were involved in supporting this work.