In-air Hand Gesture Signature Recognition: An iHGS Database Acquisition Protocol

Introduction

A conventional dynamic signature recognition usually uses a special digitized device to capture the dynamic properties of a signature. A stylus pen is used to sign the signature on the surface of the digital tablet. This leaves a subtle track, exposing the signature information to others. A forger could learn the pattern from what they obtained from the tablet surface.

Numerous acquisition approaches have been proposed to replace the usage of a tablet for dynamic signatures. For instance, two ballpoint pens with sensors to measure the pen movement during the signing process,¹ a wearable device on the wrist (i.e. smartwatches) to capture the hand motion,² or an on-phone triaxial accelerometer built in a smartphone.^3,4

The introduction of low-cost sensor cameras⁵ brings up new research opportunities for contactless human-computer interaction (HCI) in various applications such as robotics, healthcare, entertainment, intelligent surveillance, and intelligent environments.⁶ Human hand gestures and dynamic signature recognition are becoming prevalent. This work proposes a hand gesture signature recognition system with the capability to recognize the identity of a person in a touchless acquisition environment. Additionally, a public database is provided for evaluation purposes.

Some relevant research works have been conducted using their own collected database. Tian et al.⁷ introduced a Kinect-based password authentication system to explore the feasibility of a Kinect sensor to authenticate user-defined hand gesture passwords. In Ref. 8, the authors proposed a similar hand gesture signature recognition where the hand trajectory was used as the feature. The performance was evaluated on a self-collected database, consisting of 50 different classes. Empirical results demonstrated the feasibility and benefits of depth data in verifying a user’s identity based on a hand gesture signature. Fang et al.⁹ proposed a fusion-based in-air signature verification. The user’s fingertip was tracked and the signature trajectory was extracted from a video sample captured by a high-speed camera. Malik et al.¹⁰ implemented a neural network in recognizing hand gesture signatures for identity authentication. A CNN-based hand pose estimation algorithm was employed to estimate the hand joint position for the index fingertip. Multidimensional dynamic time warping (MD-DTW) was adopted to match the template and test signature data. It was tested on a self-collected dataset with 15 classes. The empirical results exhibited a promising recognition performance with the presence of depth features. Li and Sato ¹¹ proposed an in-air signature authentication using the motion sensors of smart wrist-worn devices. The system captures signal-based gyroscope and accelerometer measurements employs a recurrent neural network (RNN) to classify between genuine and imposter hand signatures of twenty-two (22) participants. The research reported a highly promising equal error rate (EER) of only 0.83%. However, this research only tested the random forgeries of the signature.

From the literature, the existing studies were mainly utilizing their self-collected databases. To the best of our knowledge, there is no publicly available hand gesture signature database. The existence of a publicly available database can provide a freely available source of data to encourage more researchers into the field. For this reason, we present an openly available database, collected by the Microsoft Kinect sensor camera. To protect the privacy of the contributors, only depth information will be shared.

Database collection

A Microsoft Kinect sensor camera is used as the main acquisition device to collect the samples of in-air hand gesture signature (iHGS) via its built-in IR projector and IR camera. A sample is a video clip that contains a set of image sequences disclosing the hand movement of a signature signing. The Kinect camera is capable of capturing up to 30 depth frames per second (fps). The number of image sequences (frames) of each sample corresponds to the duration of the hand movement and might be varying in each signature. Additionally, other computational factors such as heavy graphical processing and input latency affect the fps in each enrollment. These latencies may lead to a drop in the rate of fps, causing information loss. Thus, to ensure validation, the collected samples that have a fps rate of less than 27 are dropped/removed and the sample is re-captured through a similar procedure again. Figure 1 and Figure 2 depict the implementation of the iHGS sample acquisition process from both top and side views. The distances and spaces between the sensor camera and the subject were carefully chosen to ensure the entire body could be captured during the acquisition process. A more detailed data acquisition protocol can be found in Ref. 12.

Figure 1. The top view of iHGS sample acquisition.

Figure 2. The side view of iHGS sample acquisition.

The database is named iHGS database. The data collection was conducted in two separate sessions and the entire process took four months to complete. Samples for the second session were collected with a time interval of approximately two to three weeks from the first session. This arrangement is intended to allow the intra-variances in genuine hand gesture signatures, better reflecting real-world situations. Before enrolment, the flow of the entire enrolment process was explained to each participant. They were given ample time to practice and familiarize themselves with the process before data acquisition.

A total of 100 participants were successfully enrolled. Among the participants, 69 were male and 31 female, aged from 18-40 years. 90% of participants were right-handed (signing with their right hand) with only 10% using their left hand (left-handed). Table 1 summarizes the characteristics of the iHGS database.

There are two subsets of our iHGS database: (1) genuine dataset, and (2) skilled forgery dataset. For genuine dataset, each participant provides 10 genuine samples in each session (session 1 and session 2). A total of 2000 (10×2×100) samples were gathered for this genuine dataset.

A skilled forgery dataset contains forged signature samples. Each forger was provided with one genuine signature sample (signed by the genuine user on a piece of paper) randomly. They were asked to learn the signature with as much time as they needed. Then, each forger was asked to imitate the assigned signature 10 times. A total of 1000 skilled forgery signatures were successfully collected. However, 20 skilled forgery samples from two forgers (10 samples each) were corrupted due to the hardware error. Thus, only 980 skilled forgery samples were obtained. Table 2 summarizes the number of hand gesture signatures for the two subsets in the iHGS database.

Methods

Experimental results

Two types of performance analyses are conducted: (1) classification performance analysis, and (2) robustness analysis against forgery attacks. A well-known multiclass Support Vector Machine (SVM) is adopted in the classification analysis through a One-versus-One (OVO) approach. The genuine dataset is randomly divided into a training set and a testing set with a ratio of m:n where m is larger than n. The training set is further partitioned into two subsets: validation subset and training subset with the ratio of m_p:n_q. The training subset is to train the SVM model; while the validation subset is to find the optimal model parameters for a minimal validation error. The model is then tested on the testing set for performance evaluation. The robustness performance analysis measures the security level against impersonation attempts. It demonstrates two attacks: random forgery and skilled forgery. In the former, a testing sample that belongs to a subject i is compared with all the remaining samples of other subjects in the genuine dataset. In the latter, a forged sample of a subject j (from the skilled forgery dataset) is matched with a claimed identity’s sample (i.e., genuine subject i’s sample) from the genuine dataset.

Classification performance analysis

This analysis is implemented using the multi-class classification feature which is available in a library of SVM (LIBSVM) in MATLAB.¹⁹ The samples of the genuine dataset are randomly partitioned into training, validation, and testing subsets, refer to Table 3.

Table 3. Data distribution in SVM classifier analysis.

Genuine dataset	Training samples	1000
	Validation samples	400
	Testing samples	600
	Total	2000

A polynomial kernel of the SVM classifier is utilized as part of our machine learning model. The samples were randomly partitioned into training, validation and testing subsets to evaluate the model’s performance. For cross-validation purposes, we repeated this random partitioning process five times using five different subsets. The hyperparameters for the polynomial kernel are tuned as such that the gamma (γ) is set to 20, the degree of the polynomial (d) is set to 2 and the cost (C) is set to 1. These hyperparameters were determined through empirical testing, and the settings that proposed yielded optimal and stable performance across our multiple experiments were used. The averaged classification measurements including precision, recall, specificity, and F1-score and the standard deviation are reported in Table 4. The accuracies among features are illustrated in Figure 1.

Table 4. Performances of precision, recall, specificity, and f1-score for polynomial kernel SVM.

Feature notation	Prec.	Recall	Spec.	F1-score
Vx	63.82 ± 2.51	61.43 ± 2.00	99.61 ± 0.02	60.44 ± 2.19
VY	64.94 ± 2.60	61.20 ± 2.15	99.61 ± 0.02	60.59 ± 2.19
VXY	88.45 ± 1.14	86.63 ± 1.16	99.87 ± 0.01	86.44 ± 1.32
VHOG	93.14 ± 0.70	91.63 ± 1.05	99.92 ± 0.01	91.63 ± 1.07
VBSIF-MHI	89.53 ± 1.32	88.03 ± 1.40	99.88 ± 0.01	87.83 ± 1.45
VBSIF-x	90.50 ± 1.48	88.87 ± 1.67	99.89 ± 0.02	88.74 ± 1.66
VBSIF-Y	92.20 ± 0.91	90.77 ± 0.77	99.91 ± 0.01	90.60 ± 0.83
VBSIF-XY	97.80 ± 0.30	97.43 ± 0.35	99.97 ± 0.00	97.42 ± 0.33
VBSIF-MHIXY	94.63 ± 0.56	93.57 ± 0.63	99.94 ± 0.01	93.55 ± 0.58

Figure 1. Classification accuracies of polynomial kernel SVM.

The classification results show the two BSIF features, V_BSIF-XY and V_BSIF-MHIXY achieving the best accuracy scores of 97.43% and 93.57%, respectively. It is followed by the HOG feature V_HOG with an accuracy of 91.63%. It is noted that the system vaguely classifies the summation features, V_X and V_Y with accuracies of 61.43% and 61.20%. However, there is a boost in performance when concatenating them together, achieving 86.63% classification accuracy.

The results found that certain vector-based features such as V_BSIF-XY and V_BSIF-MHIXY, possess high levels of discriminative information for classifying in-air hand gesture signatures. Compared to other methods that involve complex preprocessing, the proposed vector-based features are extracted directly from the raw data, without the need for sophisticated techniques. These features can be used directly for classification model training, such as the SVM model, making it more convenient for real-world applications. Furthermore, the small value of standard deviation associated with these features suggests a high degree of stability in predicting hand gesture signatures. This is important in any classification task, as it ensures that the classification algorithm produces consistent and reliable results across a range of input data. The stability of these features is especially valuable in applications where the quality and consistency of the input data may vary. In summary, our findings demonstrate that vector-based features, particularly V_BSIF-XY and V_BSIF-MHIXY, offer a robust and reliable approach to iHGS classification. These features are easy to use and require minimal preprocessing, making them ideal for real-world applications that require efficient and accurate classification algorithms.

Conclusions

In this paper, we presented a self-collected iHGS database and a detailed description of the acquisition protocol to collect the database. Several basic sets of vector-based features were extracted from the samples. This paper also investigated the effectiveness of classification capability as well as the robustness against forgery attacks. The experimental results for both analyses have shown promising results with the appropriate features extracted from the samples. Our analyses demonstrate the potential of iHGS in both recognition and verification. However, there is room for future exploration in iHGS. The current database was collected in a controlled environment. As a biometric authentication, other external factors such as angles of the camera, the distance between user and acquisition devices, different background complexity, etc should be considered. In particular, it could be further extended by considering those uncontrolled environmental factors to increase the challenge of the database.

Data availability and materials

Figshare: In-air Hand Gesture Signature Database (iHGS Database) https://doi.org/10.6084/m9.figshare.16643314

This project contains the following underlying data:

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Ethics approval and consent to participate

The experimental analyses were established, according to the ethical guideline and were approved by the Research Ethics Committee (REC) with the ethical approval number EA1452021. Written informed consent was obtained from individual participants.

Author contributions

W.H. carried out the experiment with support from Y.H. and H.Y. coordinated the data collection and establishment of the database. Besides, W.H. took the lead in writing the manuscript while Y.H. and H.Y. provided critical feedback and helped shape the analysis and manuscript.