Region-wise landmarks-based feature extraction employing SIFT, SURF, and ORB feature descriptors to recognize Monozygotic twins from 2D/3D Facial Images

1.1 Motivation

Monozygotic and dizygotic twins are the two basic types of twins that exist. Monozygotic twins are the ones that result from the fertilization of a single egg that splits into two and are called identical twins. Dizygotic twins are the ones that result from the fertilization of two separate eggs and are called Fraternal Twins. The identical twin faces are depicted in Figure 1.

Figure 1. Identical twin faces.⁶

The existing automated twin recognition system does not work well for monozygotic twins. Finding differences between identical twins can be difficult due to their similar features, such as color, ears, noses, DNA, eyes, and fingerprints. In recent years, the birth rate of twins has increased along with the increase in crime.⁷ The similarities between identical twins have been taken advantage of and used for fraud and criminal activities, hence, there is an immense need for a reliable and authentic identical twin identification system. Since many twin detection methods rely on factors such as fingerprints, palm prints, speech recognition, iris, retina, mouth, ears, skin color, etc., and need significant processing time, the proposed approach bases its twin recognition on facial picture analysis. The principal objective is to create an accurate facial recognition system to identify identical twins implicated in criminal prosecution. The present proposed method is the result of an attempt to incorporate identical twin face recognition to overcome the limitations and challenges referred to Refs. 2 and 8–10.

1.2 Statement of the problem

The genetic similarities between monozygotic twins present a distinctive challenge in forensic science¹¹: when conventional identification methods fail to differentiate between identical twins in criminal investigations, judicial outcomes become highly unpredictable, potentially resulting in wrongful convictions. According to a previous study, the global twin birth rate has increased by one-third on average over the last 40 years, accompanied by an increase in crime. Research by Rehkha et al.¹² documents this trend, noting that twin births have grown from 18.9 to 33.3 per 1000 births. This presents unique challenges for forensic identification, as genetic similarities between identical twins can create substantial difficulties in criminal proceedings. When one twin is involved in a crime, the judicial outcome becomes highly uncertain. Therefore, it is important to avoid making mistakes when using biometric techniques for identification to avoid convicting someone innocent. This approach achieves high identification rates and accuracy, which makes it suitable for real-world applications such as biometric security, access control (Tribuana et al.¹³) and surveillance systems.

The current research extends previous work into identical twin recognition (Sanil et al.^8,14) by shifting focus from global feature analysis to region-specific local feature extraction in 3D facial meshes. While earlier studies established foundational approaches for twin differentiation, this work addresses specific limitations by targeting previously underutilized facial regions. The core problem statement remains consistent between these studies, as it is foundational to the research area. However, the methodology and approach applied in this manuscript are significantly different, providing new insights and extending the previous research contributions. The previous research established promising frameworks for twin differentiation using global feature extraction in both 2D⁸ and 3D¹⁴ facial images. However, low-contrast parts of the face, such as the “Cheeks,” the “front of the head,” and the “jaw boundary,” potentially containing subtle morphological differences between twins were not focused. The features extracted from these locations can capture tiny changes between identical twins that were missed during global feature extraction. The prior work has been explicitly cited in the current manuscript to ensure proper acknowledgment of earlier contributions. The distinction between the problem statement and the new methodology applied in this work justifies the originality of the current study. By incorporating region-wise landmark analysis and advanced feature descriptor algorithms, we aim to capture minute morphological variations that escape detection through global methods, thereby significantly improving the accuracy and reliability of identical twin differentiation in forensic applications.

The proposed framework for this study was inspired by the factual case history of crimes committed by identical twins. The following is the case history of these crimes. This case series highlights the significant forensic challenges arising from the genetic similarity of monozygotic twins, demonstrating critical limitations in DNA-based criminal investigations across multiple jurisdictions.

The technology behind face recognition has changed over time. Researchers initially concentrated on using 2D facial photos, however, this method had issues with head orientations and changing illumination. To address these problems, scientists started investigating 3D facial recognition. Face details like depth and curves are captured in more detail by 3D photographs than by 2D ones. But there are disadvantages to 3D methods as well. They are not feasible for real-world applications since they need a lot of file storage and processing power. More recently, researchers have discovered that a superior answer is produced by merging 2D and 3D methods. While addressing the shortcomings of each technique separately, this combination (multimodal) strategy leverages the best aspects of both approaches. Results from the 2D-3D combined methodology are more accurate than those from each technique alone. The ND Twins and 3D TEC datasets have not been used together in many studies; thus, our proposed approach will make use of both to create a novel multimodal framework.

Hence, the proposed study aims to propose a novel and effective approach to developing an accurate 2D/3D facial recognition system based on local characteristics that mimic the method used by forensic experts to identify identical twins implicated in criminal acts, which are deemed difficult due to gene similarity, rendering standard forensic DNA testing ineffective using ND twin and 3D TEC datasets.

1.3 Paper organization

This work is divided into the following sections. A survey of similar works and the literature is included in Section 2. The contributions to the research are given in Section 3. Section 4 describes the method for analysing facial images to detect local facial characteristics using machine learning methods. Section 5 describes the research methods used in this study. Section 6 presents the results of this study. Section 7 contains the conclusions and future research directions.

2.1 Background theory

It is well known in face recognition research that one of the biggest problems is differentiating similar faces, particularly lookalikes and identical twins. The task is further complicated by the striking similarity in the biometric prints of the faces of identical twins. To increase the accuracy of face recognition, numerous sophisticated algorithms and databases have been created over time and tested in a variety of scenarios. However, these initiatives have often failed to meet expectations. Improving the present automatic facial recognition systems are necessary due to the increasing number of identical twin births and their increased involvement in fraud and criminal activity. Since forensic face recognition techniques are made to comply with legal procedures, these systems must be integrated with them.

Face anthropometry: The scientific examination of human body dimensions and proportions is known as anthropometry. Morphology is the study of forms in two and three dimensions, with an emphasis on the quantitative analysis of form and size that arises from the combination of geometry and biology. The face recognition system was developed in the 18th century by French biometrics researcher and police officer Alphonse Bertillon, who applied the anthropological method of anthropometry for criminal identification to create a break- through system. To describe human faces, qualitative and quantitative aspects were used. Figure 2 shows the anthropometric landmarks for the frontal facial picture (a) and the side profile of the face (b). The definitions for these landmark points are provided in Table 1. Table 1 provides a list of sample anthropometric landmarks on the face, along with their corresponding descriptions. These landmarks are used to derive various facial ratios. This comprehensive table provides definitions for key anthropometric landmark points, detailing their precise anatomical locations. These points are essential for accurate face measurements. The landmarks cover points on the face to provide a systematic approach to human face measurement. This study focuses on the analysis of facial characteristics and highlights the critical role of local regions¹⁸ in face recognition systems.

Figure 2. Representation of anthropometric landmark points on [a] Frontal facial image and [b] Side posture of the face.⁸

2.2 Literature review on feature-based face image analysis for identical twin recognition

Over the last decade, researchers have made important breakthroughs in the field of face recognition of identical twins. Several researchers investigated various methods to this problem which are listed here. Biometric methods are of interest because they can distinguish between faces that are similar to each other, as mentioned in.^19,20 The identification of criminals in the forensic sciences is one area in which face recognition systems are heavily utilized. Face recognition has several difficulties while being widely used and having many useful applications.²¹ says that humans are naturally good at telling people apart based on their distinctive facial features. The human ability to measure a face for identification has been the main focus of facial recognition research. Consequently, there is a great deal of attention on creating new algorithms that imitate “human vision” to identify faces. Compared to studies on identification methods for regular people, there are far fewer studies on recognition technologies for identical twins based on various biometrics such as face, fingerprint, and iris. The study focuses significantly on work that discusses face recognition methods relevant to identical twins, similar faces, and their applications, more so to forensic aspects. Recent studies have shown that since identical twins share similar features due to their genetic similarity, the existing automated twin recognition system does not work well for monozygotic twins.^22,23 Identical twins have similar features such as colour, DNA, eyes, ears, noses, and fingerprints, and therefore, differentiating the identical twins is a challenging task.^24,25 The technologies currently used in the field of feature-based facial image analysis to identify twin faces, and similar faces are reviewed in the literature study that follows.

Moung et al.¹ proposed advanced techniques to address challenges in face recognition, including (i) automated face detection, (ii) variations in facial pose angles, (iii) occlusion impacts, (iv) diverse facial expressions, (v) aging-related changes, (vi) varying conditions of lighting, (vii) low image resolution, (viii) similarities between the identical twins and look-alikes, and (ix) other technical constraints. In addition to face recognition technologies, several biometric approaches have been proposed to improve the verification of the identification of identical twins.

Kukharev et al.² presents a review that covers the following topics: morphometry; unique cases in face image recognition, such as identical twins and lookalikes; both qualitative and quantitative techniques for determining parameters and assessing facial features; as well as multiple approaches using digital anthropometry. It also provides a brief history of the development of anthropometry in contemporary techniques and strategies that use computer technology to measure facial anthropometry. Primarily theoretical; lacks empirical validations.

Nafees et al.⁶ used a “Gray-level co-occurrence matrix” along with a “Haar-Cascade classifier” to assess and discriminate against identical twins by studying RGB histograms of the eyes, lips, and face. One of their twin recognition investigations was carried out on a small dataset of just five pairs of twins. They added that this planned research on 3D face recognition can also be tested in controlled environments that can accommodate a variety of facial variances. A limited sample size is used.

Sanil et al.⁸ introduced a unique method that combines human knowledge and machine learning to produce forensic evidence using 2D facial photos collected from the Web. To discover related facial traits, their solution uses machine learning approaches combined with hyperparameter tuning. For 2D facial photographs, the Euclidean distance is utilized to calculate the straight-line distance between two positions. In the case of curved surfaces, the geodesic distance is calculated by adding the linear distances between adjacent landmarks along the facial curvature between the two locations. Their technique achieved 78% accuracy with a small dataset. Low-contrast areas of the face, such as the “Cheeks,” “Forehead,” and “Jaw boundary,” were not focused, which could catch subtle changes between identical twins. A limited sample size is used.

Rehkha et al.¹² used different multimodal biometric techniques, such as hair wrinkles, facial marks, and facial features, using the PCA algorithm to discriminate between identical twins. When there are few biometrically comparable traits between twins, such as identical occipital hair whorls, they can be distinguished from one another more effectively by looking at their external ears and determining whether they are left- or right-handed.

Asogwa et al.¹⁵ demonstrated an innovative technique that uses machine learning algorithms to identify identical twins and distinguish between two similar suspected faces belonging to distinct identities. This system is intended to aid in international criminal investigations. To further improve the system’s processing speed and recognition accuracy, more machine learning or deep learning-based approaches can be applied. Lacks specific performance metrics.

Biswas et al.²¹ carried out various experiments under varying conditions to develop algorithms that mimic the human ability to discriminate facial features in twin identification. The results of this investigation lead to the understanding that the use of facial marks, along with the existing set of features, improves existing face recognition algorithms and machine performance. Their work recorded a precision of 78.825%. Limited feature integration strategy.

Phillips et al.²² published a study on identical twins considering a time lapse of one year using the ND-Twins dataset. It was the first comprehensive analysis of twins’ faces utilizing three of the best submissions for the Multi-Biometrics Evaluation (MBE) COTFRS (commercial off-the-shelf face recognition system). It was revealed that due to sex, age, variations in lighting conditions, and expressions, the precision was greatly reduced. They also presented performance curves and error rates for various algorithms of face matching for differentiating identical twins under variable situations, which includes even those images taken with a time-lapse of one year and differentiating by gender and age. Performance degraded with lighting and expressions time lapse. Paone et al.²⁶ considered testing 7 different algorithms under various conditions for face recognition. Performance is assessed considering the four covariant: (i) age, (ii) gender, (iii) expression, and (iv) illumination. Their results revealed that they needed studio-like ideal conditions concerning illumination and facial expression with images acquired within a time-lapse of one year (not 2 years apart). It also showed that the performance of the algorithms was not affected by age and sex. The result ranged from 4.1 to 17.4 % as the best equal error rate.

Mousavi et al.²⁷ suggested a modified SIFT (M-SIFT) method in conjunction with crowdsourcing to discriminate between identical twins. They divided each facial image into five regions: eyebrows, eyes, nose, mouth, and face curves. Of these regions, the face curve was found to be the most important feature to differentiate between identical twins. Using this technique, 650 pictures were gathered in total, 115 identical twin pairs and 120 nonidentical twin pairs. As demonstrated by the test results, the lowest Equal Error Rate (EER) to identify identical twins was 7.8% for the full image, 8.1% for frontal images exclusively, and 10. 1% for PAN motion images. However, the facial region landmark detection (FRLD) method was unable to identify the landmark regions.

Sudhakar et al.²⁸ developed a fusion-based technique to differentiate identical twins. Principal component analysis (PCA), Gabor distance between face components, local binary patterns (LBP), and histogram-oriented gradients (HOG) were used in this study to extract features, which were then merged. Based on the scores produced by this fusion, twins were found. The best features were chosen using particle swarm optimization, and the images were trained and tested using a support vector machine (SVM). Compared to earlier techniques, this method produced greater precision and required less processing time. However, only photos with different faces and stances were considered; realistic photos were not.

Afaneh et al.²⁹ introduced a two-level decision process-based technique for identical twin recognition. They combined a CNN with fusion at the score, feature, and decision levels to increase accuracy. The study used ND TWINS-2009-2010 and traditional FERET data and used feature extractors, including PCA, LBP, and HOG. The results of the experiment demonstrated that the multimodal biometric system outperformed the unimodal systems in recognition. Under regulated illumination, the system achieved an Equal Error Rate of 2.2%, while for neutral expressions, it achieved a rate of 2.7% for identical twin recognition. Performance was highly dependent on controlled lighting.

Ahmad et al.³⁰ to distinguish between identical twins put forward a deep neural network. They used triplet loss to implement two different CNN models. Even powerful deep networks find it difficult to recognize identical twins with a precision of 87. 20% that they require.

Nahar et al.³¹ applied a Transfer Learning approach in their study that incorporated geometric and photometric features. The authors also evaluated two networks trained with VGG-16. The integration of geometric and photometric features resulted in a precision of 98%. However, additional imaging data of identical twins is required. Four sets of twins, each recorded in 17 distinct positions, are included in Google Data. Photometric characteristics alone yielded an accuracy of 96%. In addition, different transfer learning techniques can be integrated with other modalities such as speech, facial recognition, and palm prints.

Venkatesan et al.³² worked on a combination of Mean Landmark Points (MLPs) algorithm, SVM, and 3D PCA to present a face recognition system. Then, it was observed that the use of a carefully selected training dataset, along with the application of an SVM classifier on extracted features, could significantly improve the recognition rate. It was also noted that a process of initially segmenting multiple regions of the face, followed by each region being classified separately, and thereafter performing a fusion, would enhance the accuracy. It is also observed that this accuracy can be improved by using a multiclass SVM.

Parde et al.³³ investigated the ability of people and a DCNN to discriminate between identical twins and faces that are similar from different angles. They evaluated 87 individuals' ability to determine whether two faces in a group belonged to the same person or someone else using photos taken from frontal to profile views. According to the study's findings, DCNNs are becoming increasingly accurate in difficult face-identification scenarios, which could help applications in security and forensic settings by fostering human-machine cooperation.

2.3 Literature review on Facial Image Analysis to Identify Identical Twins using 3D-TEC dataset

Sanil et al.¹⁴ highlighted the challenges faced in distinguishing identical twins due to their genetic similarities. The system utilizes geodesic distance algorithms such as Dijkstra’s algorithm, the Fast-Marching method, and A* algorithms for GD computation on 3D images, which account for the curvature of the facial surface, providing a more accurate measure of distances between 3D facial landmarks using the 3D-TEC dataset. This approach achieved 90% accuracy on a limited dataset, but their reliance on small sample sizes raises questions about generalizability to larger populations. While their real-time geodesic distance computation showed promise, the research failed to address computational complexities when scaling to larger datasets.

Vijayan et al.³⁴ conducted pioneering work with 3D twin images using SIFT and ICP algorithms but struggled significantly with expression variations (Cases III and IV), revealing fundamental weaknesses in handling changes in facial expression between twins. They used 3D facial landmarks including geodesic distance measurements, however their results showed that 3D techniques still had limitations, with recognition rates of about 72%.

Cai et al.³⁵ Pre-ResNet variants with multiscale triplet loss supervision reached 94.07% accuracy, but their approach depends critically on precise detection of the nose tip and pupils. This creates a significant vulnerability point, as recognition accuracy deteriorates substantially with even minor landmark detection errors. To obtain state-of-the-art recognition results, Quy et al.³⁶ introduced a SIFT-based method that achieved 84.6% accuracy but required parameter adjustments specific to the template database, limiting its adaptability to new datasets and real-world scenarios where parameter optimization may not be feasible. Required parameter tuning for the specific database.

Kim et al.³⁷ proposed a 3D face recognition system that utilizes a deep CNN. VGG- Face is a network that is optimized for 3D data after being pre-trained on 2D face images. The adaptation of VGG-Face showed substantial performance degradation (dropping to 80%) when handling expression variations, indicating limited robustness to emotional expressions that would be common in real-world applications. Compared to Cases I and II, the rank-1 performances in Cases III and IV were quite low (79.9% to 81.3%).

Al-Osaimi et al.³⁸ introduced an innovative method for expression-invariant 3-D face recognition, which uses the rotation-invariant and adjustable integral kernel (RAIK) technique to create key points for matching. The RAIK approach achieved 89% accuracy with improved expression invariance but relies on a heuristic graph search that introduces unpredictability in edge cases and potentially limits computational efficiency for large-scale deployment.

Li et al.³⁹ presented multiscale and multicomponent local normal patterns (MMSMC- LNPs), a unique approach for 3D facial recognition. MMSMC-LNPs approach with weighted sparse representation reached 95% accuracy but required a specialized training set, raising concerns about performance consistency across different demographic groups and data collection conditions.

Gilani et al.⁴⁰ developed the first deep three-dimensional face recognition network (FR3DNet) trained in 3.1 million 3D facial scans of 100,000 identities. The largest data set was known as LS3DFace. It incorporates information from several difficult public datasets, such as ND-2006, FRGC v2, Texas-3D, Bosphorus, GavabDB, BU4D-FE, CASIA, BU3D-FE, UMBDB, and 3D-TEC. FR3DNet, despite achieving 98% precision, depends on complex 3D face-dense correspondence for data augmentation that cannot guarantee identity closure properties. This introduces label noise that may compromise the model’s reliability in forensic applications where certainty is paramount.

Dutta et al.⁴¹ presented a unique mathematical model to break down images in the range face into complementary components. The multistage approach with genetic algorithms and CNN classification suffers from excessive complexity, making practical implementation challenging and potentially limiting transparency in decision-making, a crucial concern for forensic applications.

Chen et al.⁴² investigated the hybridization of feature extraction approaches by mixing SIFT with deep learning architectures. Their findings suggested that hybrid techniques could detect small changes between identical twins more effectively than single-method approaches.

In summary, it has been observed from reviews that existing facial recognition systems do not effectively identify identical twins. According to the current review level, facial recognition technology cannot successfully discriminate between identical twins. Due to their extreme genetic similarities, identical twins cannot be distinguished from each other using traditional forensic DNA tests, making the process nearly impossible.²⁰ Even for deep neural networks, it is difficult to identify identical twins based on facial photographs in an uncontrolled setting.³⁰ Despite the researchers’ efforts to create a more accurate and realistic twin detection system, there is still potential for advancement to overcome the aforementioned restrictions by taking into account a larger feature set to allow for fair twin discrimination. As a result, there is increasing interest in differentiating between identical twins using different biometric attributes and methods, particularly in forensic-related fraud and crime. As a result, the primary goal is to focus on low-contrast areas of the face, such as the “Cheeks,” “Forehead,” and “Jaw boundary,” and to analyze them using various approaches through a quantitative approach, which aids in identifying the performance measures that have a greater impact on the model. The ND -TWINS - 2009-2010 data set and 3D TEC were discovered to be the most difficult of the few data sets utilized for identical twin recognition, which is considered in our proposed approach.

4.1 Overview of the proposed approach

Gathering the images from the ND twins and the 3D TEC database is the first stage. Pre-processing of the chosen image is required to recognize and crop the faces of each member of the twin pair.⁴³ The exact location of 468 points on 2D/3D facial photos was achieved using the media pipe framework.⁴⁴ Next, a region-wise selection of landmarks is performed using 468 landmarks to determine the number of significant points in the selected region, such as the nose, lips, eyes, eyebrows, and facial curves. These regions can serve as the basis for local feature extraction. The region-wise feature extraction is then performed by implementing the SIFT, SURF, and ORB feature descriptors on a single image. A list of key points with valuable image descriptions is achieved. Key points are areas of interest, which means that when a human sees an image at that particular moment, he observes certain aspects of that image. The term “Key points” refers to the specific spots of interest that a machine identifies when it examines an image. Descriptors are arrays or bins of numbers. These are used to describe the features; using these descriptors, we can match the two different images. The key points and image descriptors are computed for each detector. The system separates important points into two categories: mismatched points and matched points. The similarities between the two images are emphasized by the match points. Mismatched points serve as appropriate visual aids for illustrating image disparities. The region-matched and mismatched key points are used to calculate the ratios (16 * 3) concerning facial regions such as the eyes, eyebrows, nose, mouth, and face curves using SIFT, SURF, and ORB. The findings of these three descriptors are combined to create a fusion of characteristics. To determine how similar or distinct twin faces are depending on their attributes’, fused features are analysed employing machine learning models. A range of machine learning techniques, such as Nearest Centroid classifiers, eXtreme Gradient Boosting, Light Gradient Boosting Machine, and Support Vector Machine, were used to classify the data in the proposed experiment. The best machine learning models were chosen by comparing the efficacy of different models for a data set using the lazy-predict package. Comparison research has been illustrated using metrics such as the area under the curve (AUC), false positive rate (FPR), and true positive rate (TPR).

The proposed facial recognition system is divided into several steps, including pre-processing, face and landmark identification, extraction of features, and fusion, classification, and making decisions. Figure 3 displays the block diagram for the proposed methodology.

Figure 3. Schematic illustration of the Local feature extraction approach.

5.1 Data collection and validation

The primary step in facial image analysis is image acquisition. Data collection is done in the initial stage of the research before achieving the objective. The details of the data sets required in the proposed method are expressed below. In the proposed study, ND-TWINS-2009-2010 Dataset⁴⁵ and the 3D Twins Expression Challenge (3D-TEC) data set⁴⁶ were used.

Figure 4. Sample facial images of ND-TWINS dataset.⁴⁵

Figure 5. Sample facial images of 3D TEC dataset.⁴⁶

The data set’s picture representation format is JPEG and PNG compatible. Of the images in the data set, 20% was used for testing and 80% for training. The ND-TWINS and 3D TEC datasets were chosen for their high-quality photos, variety of facial features, and importance to the identification of monozygotic twins. The ND-TWINS dataset contains high-resolution 2D facial photos with lighting, position, and expression variations, making it ideal for testing feature extraction and classification methods. Meanwhile, the 3D TEC dataset has 3D facial scans, allowing for a more comprehensive analysis by including depth information, which improves robustness in difficult conditions.

5.2 Facial image analysis

The process of extracting useful information from pictures or videos of people’s faces using computer vision algorithms is known as facial image analysis. It analyses faces in images, videos, and real-time situations using computer algorithms and machine learning. Both human observers and computer systems can deduce a great deal from facial photographs, including age, identity, gender, race, emotions, and even attractiveness. The potential uses of machine learning-powered facial analysis methods have received much attention lately. Several important areas, including face detection, facial feature recognition, facial identification, and facial photo interpretation, are the focus of ongoing research in this field. The ability to identify and quantify the locations of facial feature points within an image is essential for performing face analysis tasks. The study of facial image analysis promises more advancements in comprehending and utilizing the complex information that human faces express as methods continue to develop. Figure 6 presents the results of a manual study of 468 anthropometric points in various facial regions.

Figure 6. Manually mapped 468 facial landmarks.

Figure 7. Face detection with 6 landmarks using MediaPipe framework considering the twin images from the ND twins dataset.

Figure 8. Face detection with 6 landmarks using MediaPipe framework considering the twin images from the 3D TEC Dataset.

Figure 9. Landmarks detection using MediaPipe framework considering images from the ND Twins Dataset.

Figure 10. Landmarks detection using MediaPipe framework considering images from the 3D TEC Dataset.

The output of superimposing 468 landmarks on facial image from the ND Twins dataset, creating a face mesh using the 468 facial landmarks extracted and highlighting the facial regions are shown in Figure 11.

Figure 11. Superimposed 468 points, facial mesh is generated based on the extracted landmarks, and the facial regions are highlighted.

In the current study, the MediaPipe framework face mesh solution was used to annotate the 468 landmarks in any uncontrolled condition (including the entire forehead region and additional points in the jawline region) from which significant landmarks were selected to obtain an increased number of variances that improved recognition accuracy. Forensic experts in criminal investigations rely only on anthropometry-based landmark measurements that are widely accepted in the court of law as statistical evidence; therefore, using a deep-learning-based approach is not acceptable. Hence, the MediaPipe framework, which works well even in real-time images, is used in this approach for landmark detection. This method is effective in real time for a range of illumination conditions, faces that are hidden, and faces that are of different sizes and orientations. The facial mesh topology provides more information than is needed; therefore, one can select the information that is needed. The MediaPipe framework provides more information than needed, and it is also possible to select only essential information with real-time performance. Applications currently implemented with MediaPipe include face detection, face mesh annotation, iris localization, hand detection, pose estimation, hair segmentation, object detection, tracking, and three-dimensional(3D) object detection.

A series of points found there can be used to identify each of the five regions:

Figure 12. Region-wise selected landmarks.

5.3 Region-wise landmarks-based feature extraction

The extraction of features is a fundamental task of face recognition algorithms. Because they transform visual information from an image into a numerical format that machine learning algorithms can compare and interpret, feature descriptors are crucial. The most important parts of an image are the highlights, dark areas, corners, and edges since they are always visible despite variations in the image’s form, brightness, or noise level. Identifying and describing key facial features that are useful for tasks such as facial landmark identification, facial emotion analysis, and face recognition is the aim of local feature extraction based on facial features. The objective is to isolate characteristics on the face that do not change with changes in lighting, posture, or expressions. Local feature-based extraction methods such as ORB (Oriented FAST and Rotated Feature Transform), Scale-Up Robust Features, Rotated BRIEF, and Scale-Invariant Feature Transform are widely used in computer vision for tasks such as object recognition, picture matching, and 3D reconstruction. The main goal of these techniques is to identify and characterize important spots or areas in an image that remain unchanged when subjected to various manipulations, including rotation, scale, and lighting adjustments. The proposed approach incorporates a local feature extraction technique that extracts features from local areas such as the eyes, nose, eyebrows, and face curve, among others, using well-known image matching techniques like SIFT, SURF, and ORB, both separately and in various combinations that utilize selected landmarks. A list of key points with valuable descriptions is achieved.

Using SIFT, SURF, and ORB, the regions chosen based on the landmarks are used to identify unique focal points in both images and calculate their descriptors, which represent the local image content surrounding those focal points. The two sorts of points that make up the SIFT/SURF/ORB algorithm’s essential points are matched and mismatched points. First, a twin pair’s similarity is indeed represented by the matched points. Every other key point, except for match points, is a local maximum; no two points share a similar description vector. These are referred to as mismatch points. The essential points that are out of order serve as an appropriate illustration of the twins’ differences. A significant piece of information on the primary difference between two identical twin photos can be found when the concentration of mismatched sites is large.

Figure 13. The application of SIFT feature descriptor to seven facial regions.

Figure 14. The application of SURF feature descriptor to seven facial regions.

Figure 15. The application of ORB feature descriptor to seven facial regions.

Equations 1 and 2 are utilized to compute the ratio of match and mismatch points on the seven landmark regions relative to the remaining areas in the face. Ratio-based features extracted using the matched and mismatched points on the seven landmark regions:

How many matched points are found in the seven landmark zones that have been suggested is shown by the results of Equation 1. Quantifying the number of mismatched points inside the seven designated landmark zones is possible by utilizing the results of Equation 2. In case two identical twins have identical faces, the most distinguishable area between them is the one that has the greatest number of mismatched keys. The comparison of the quantity of mismatched key points in the selected landmark regions will be used to describe this. Two different feature sets were extracted and compared to find the best approach to recognizing faces. The first method uses the distances between the selected landmarks, while the second method employs ratios of distances. In this and other landmark regions, several features are proposed and extracted to evaluate the effectiveness of the face region in detecting identical twins. Equations 3 and 4 are used to obtain the following 14 features region-wise:

Similarly, using the landmarks identified by the media pipe structure, a total of (16*3) features are extracted in this study from other chosen facial landmark locations, such as the eyes, nose, mouth, and eyebrows, based on the key points generated by the SIFT/SURF/ORB algorithm. The 16 different measurements were selected to capture key facial proportions across different regions without collecting redundant information. By using three different feature detection methods (SIFT, SURF, and ORB), the system benefits from each method’s unique strengths - SIFT handles different scales well, SURF processes quickly, and ORB manages rotated faces effectively. Through careful testing of various measurement combinations, researchers found that using exactly 16 measurements with these three detection methods created the best results. These features extracted from the SIFT, SURF, and ORB are merged to form a comprehensive representation of the face, and the resultant feature vector is saved in the databases, which is used for similarity measurements.

5.4 Implementation details

The algorithm is implemented with Python 3.8 in the VS code framework and the power shell window. For automated face detection and landmark detection, existing models that are available in the “MediaPipe” framework are used. The experimental software environment was TensorFlow 2.10,⁵⁶ Python 3.11⁵⁷ and the computer configuration was as follows: 11th Gen Intel(R) Core (TM) i7-12650H 2.30 GHz; 16.0 GB RAM; PCd AMD Ryzen 1600, 16GB RAM, RTX 4070; and Cuda11.3.1 with Cudnn8.2.1. Algorithm 1 explains the feature extraction process of the proposed approach. The data was split into training and test sets using an 80:20 ratio for all experiments.

TensorFlow is an open source software library for high performance numerical computation that can be accessed here (https://pypi.org/project/tensorflow-gpu/2.10.0/).

Python Software Foundation License, which is an OSI-approved open-source license, meaning it’s free to use, modify, and distribute that can be accessed here (https://www.python.org/downloads/release/python-3110/).

The resultant ratio-based features obtained from the twin images are stored and then used for comparison through various machine learning tools such as support vector machine, light gradient boosting machine, XGBoost, and nearest centroid classifiers to find the mismatch.

5.5 Classification using machine learning algorithms

Giving an image a class through classification helps facial recognition. A class is then allocated to that similar group once the picture dataset is categorized based on the features that were retrieved from the photos. The test photos are assigned a class name by the twin recognition system after the features retrieved are compared with the training dataset. Train a classifier using the fused local features as input and a label indicating whether the faces belong to the same twin pair or not. To test the trained classifier model, extract the SIFT, SURF, and ORB features from a questioned image and enter the query object image. To identify the mismatch, the resulting ratio-based characteristics of the twin photos are saved and subsequently compared using a variety of machine learning techniques, including the support vector machine, light gradient boost machine, XGBoost, and closest centroid classifiers. Lazy Predict, a Python-based code,⁵⁸ is used to select the best machine learning models. In the proposed approach, four classification models were selected. It is useful for comparing basic models and determining which model performs best without any parameter adjustment. In this study, the four best classification models were chosen using the Lazy Predict Python program. SVM excels at finding the subtle differences between nearly identical faces and performs well even with a limited twin dataset. LGBM processes complex facial measurements quickly and efficiently, which is essential when analyzing multiple facial regions simultaneously. XGBoost helps manage inconsistencies in facial feature detection and avoids overfitting to specific twin examples, instead of learning true distinguishing characteristics.NC is useful for classifying data based on feature similarity, making it a simple yet effective approach. These classifiers were selected after evaluating their performance and suitability for the extracted features proposed compared to other traditional methods. The testing showed that these classifiers work better together than the other approaches we tested. Their combined abilities create a system that can reliably detect minor differences between twins under various real-world conditions. The accuracy, sensitivity, specificity, etc., were also measured using the ND twins and 3D TEC dataset to test the model’s performance. The Python module Scikit-learn is used to perform machine learning analysis⁵⁹ to classify facial images based on their similarity. The efficacy of the local feature extraction strategy for twin face recognition is tested using key performance criteria such as precision, recall, and F1 score.

5.6 Performance metrics for classification

The parameters of accuracy, precision, recall, F1 score, and area beneath the receiver’s operating characteristic (ROC) curve were used to compare and assess the suggested approach. A confusion matrix is an essential tool for determining a model’s benefits and drawbacks. The performance of a classification model in machine learning and statistics is evaluated using this table. The sklearn metrics module provides the Confusion Matrix, which is used to measure the number of accurate and inaccurate predictions in addition to other metrics such as precision, recall, and accuracy.

To compute all the measures above, the confusion matrix (CM) that includes true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) is shown in Table 2. The performance of classifiers utilizing supervised machine learning techniques is often evaluated using metrics obtained from the confusion matrix, which are mentioned below.

Where TP and TN stand for correct predictions, FP and FN stand for wrong predictions.

5.7 Statistical analysis of feature distributions

To explore and validate the statistical significance of differences between the two classes (same person vs identical twin), a comprehensive statistical analysis was conducted on all features in the dataset used to classify whether two facial images represent the same person (label = 0) or identical twins (label = 1).

Descriptive Statistics: For each feature, descriptive statistics were computed separately for each class as shown in Table 3. These included Mean and Standard Deviation to describe central tendency and variability. Also 95% Confidence Intervals (CI) for the mean, calculated using t-distribution-based methods, to estimate the range within which the true population means is expected to lie with 95% certainty.

Tests for Statistical Significance: To rigorously assess whether these differences were statistically significant, two tests were performed for each feature:

The statistical tests revealed that several features exhibited highly significant differences between the classes, with p-values well below 0.001 in both t-tests and Mann-Whitney tests. The discriminative top 10 features are included. These features had both large differences in mean values, and non-overlapping confidence intervals, indicating strong separation between the two classes. Boxplots were generated for the top features show in Figure 16, visually confirming that features associated with mismatch scores tend to be significantly higher for identical twins than for the same person. This validates that the classifier can capture subtle but consistent biometric differences between these groups.

Figure 16. Boxplots for the top 10 statistically significant features.

The incorporation of statistical measures and hypothesis testing provides strong evidence that certain facial features differ significantly between identical twins and the same person. These findings support the reliability of the selected features for classification tasks and justify their inclusion in any downstream machine learning models.