Adaptive Phoneme State Learning Architecture for Enhanced Speech Recognition Using Backpropagation Neural Network and Hidden Markov Model

2.1 Historical foundations

The roots of phonetics trace back to as early as 500 BC on the Indian subcontinent, with Panini meticulously describing the place and manner of articulation of consonants in Sanskrit.⁶ The chronicles of speech recognition date to 2002, culminating in a final output release in 2005, functioning proficiently across three languages: English, Spanish, and Mandarin.⁷ Operating at a speech rate of 10 Hz with a recording precision of 96 kHz/24 bit, this innovation marked a pivotal milestone. Fast-forward to 2019, another speech synthesizer emerged during the “Blizzard challenge”,⁸ pronouncing 1200 phonetic utterances at a frequency of 1.5 Hz. These early developments established the foundational principles of acoustic modeling and phonetic analysis that continue to inform contemporary speech recognition research.

2.2 Traditional HMM-Based speech recognition systems

Several researchers have contributed to the advancement of HMM-based speech recognition systems, as summarized in Table 1. These studies demonstrate various approaches to phonetic segmentation, speech synthesis, and recognition across different languages and acoustic conditions. While these prior works established HMM as a viable approach for speech recognition across multiple languages and acoustic conditions, they exhibit notable limitations, including moderate accuracy (ranging from 61.5% to 89%), language-specific implementations that limit cross-linguistic applicability, and limited handling of diverse speech qualities and acoustic environments.

2.3 Motivation for hybrid HMM-Neural approaches

Traditional HMM systems provide explicit temporal structure and interpretability through their state transition probabilities and emission distributions. However, their acoustic modeling capacity is constrained by the conditional independence assumption of observations given the hidden state. Neural networks, conversely, excel at learning complex, non-linear feature representations from raw data but lack inherent temporal constraints and explicit alignment mechanisms. This complementary relationship has motivated the development of hybrid architectures that combine the strengths of both paradigms. Hybrid models combining generative components (such as Gaussian Mixture Models) and discriminative components (such as neural networks) have demonstrated superior performance on specialized tasks where pure end-to-end models struggle due to limited training data.³⁹ This finding motivates our HMM-BPNN hybridization, which similarly combines probabilistic temporal modeling with neural discriminative power while preserving the explicit state alignment that HMMs provide. Recent comparative studies of temporal modeling architectures for noisy speech recognition have shown that hybrid CNN-LSTM models achieve 90.72% accuracy in noisy conditions, outperforming standalone CNN (90.12%) and LSTM (86.12%).⁴⁰ However, these recurrent and convolutional architectures lack explicit phoneme-state alignment and do not provide interpretable confidence scores at the phoneme level—a gap our HMM-based approach with APSL directly addresses.

2.4 Research gap and contributions of the present study

Despite the advances in hybrid HMM-neural systems documented in the literature, existing approaches treat neural network outputs as static observations that are integrated with HMM probabilities using fixed interpolation weights that do not vary based on input characteristics. No existing work dynamically adjusts HMM transition or emission probabilities based on neural confidence scores in an online, utterance-specific manner. Furthermore, most hybrid systems do not adapt their temporal analysis windows in response to classification uncertainty, potentially wasting computational resources on clear speech segments while providing insufficient acoustic context for ambiguous phonemes. Against this backdrop, the present study introduces the Adaptive Phoneme State Learning (APSL) algorithm, which integrates several key innovations: (i) Labeling synthetic waveforms with distinct features to improve phoneme discriminability during model training, (ii) MFCC-based dynamic feature extraction employing filtering to extract feature coefficients as an energy measure for robust acoustic representation, (iii) Bidirectional spectral training addressing the challenge of insufficient training observations in HMM models by encompassing both forward and backward training spectral features, introducing time-dependent windowing factors to reduce memory requirements and optimize likelihood summation across all states and (iv) Confidence-driven adaptive refinement that dynamically adjusts HMM state transitions using BPNN posterior probabilities, with dynamic window extension (Δt = 5 ms) for low-confidence phonemes to provide additional acoustic context where needed most.

The proposed APSL-BPNN-HMM model demonstrates robust recognition accuracy even in noisy environments, as validated through extensive experimentation across multiple speech corpora described in subsequent sections.

3.1 Speech acquisition

Raw speech signals are acquired through microphones, online audio files, or audio CDs. Accurate sampling frequency configuration is critical before recording. For example, a 100-second audio file sampled at 44100Hz yields $44100\times 100=\mathrm{4,410,000}$ samples, ensuring CD-quality audio. Based on Nyquist’s theory,¹⁸ the sampling rate must be at least twice the maximum signal frequency to avoid aliasing. For instance, a 10,000 Hz signal requires a minimum 20,000 Hz sampling rate. Sampling frequency selection involves a trade-off between audio quality and memory consumption: lower frequencies reduce memory usage but compromise quality, while higher frequencies enhance fidelity at the cost of increased storage. The optimal balance depends on application-specific requirements.

3.2 Voice activity detection (VAD)

Voice Activity Detection (VAD) comprises two stages: noise removal and speech pause detection. Noise elimination employs a Training-Based Noise Removal Technique (TBNRT),¹⁹ utilizing a corpus of noise types from white to environmental noise. Noise segments matching the noise dictionary are removed using high-pass and low-pass filters. Endpoint detection utilizes algorithms based on energy variance, pitch modulation, zero-crossing rate, cepstral parameters, or linear prediction coding (LPC).²⁰ VAD applies the min/max energy threshold (ET) paradigm. For sample $S_{B_i}$ in each speech segment $B_i$ , ET is defined at indices $x$ and $y$ , where $x$ represents the total signal duration and $y$ represents the duration within block $B_i$ . $S_i$ denotes the speech signal in each segment, where $S=\{1,2,\dots ,n\}$ .

Step–1: The energy is calculated using Equation (1):

Step–2: Voice Activity Detection (VAD - Equation 2)

Step–3: When $T_m$ is reached, the signal breaks until the next $T_m$ is reached.

VAD extracts speech features every 5-40 ms and compares them to base threshold $T_B$ . Features exceeding $T_B$ yield VAD = 1 (speech present); otherwise VAD = 0 (no speech). Initially assuming a 40 ms segment contains no speech, we analyze frames of 60 samples (6 ms duration) collected at 70 kHz. The average threshold for each frame is determined using Equation (3):

Since loudness varies among speakers, we focus on minimum loudness. Using Praat,²¹ we analyzed loudness ranges to categorize $T_m$ , employing a Python script to eliminate signals at the $T_m$ threshold. For instance, the quietest sound measured 59.3 dB, with quiet segments ranging from 59-62 dB. The first segment below $T_B$ is designated as $T_m$ . Speech typically begins softly, peaks at maximum $T_M$ , then decreases, defining the minimum-maximum energy range. The quiet threshold is set at -25.0 dB, with segments below classified as quiet. Temporal constraints include a minimum pause duration of 0.1 seconds between words (longer for sudden loud sounds; shorter durations are not classified as quiet) and a minimum sounding time of 0.05 seconds (representing inter-syllable pauses).

3.3 Feature extraction

Feature extraction techniques include mel-frequency cepstral coefficients (MFCC),²² vector quantization (VQ),²³ artificial neural networks (ANN),²⁴ Hidden Markov Models (HMM),²⁵ and dynamic time warping (DTW).²⁶ This study employs MFCC for framing and HMM for windowing. MFCC-based feature extraction involves two steps: Pre-emphasis and Framing.

Pre-emphasis : High-frequency sounds typically have lower magnitudes, leading to higher distortion and compromised speech quality. Pre-emphasis counters this by suppressing high-frequency components and boosting magnitude, producing a smoother profile than the original audio. The pre-emphasis factor $\alpha$ is calculated using Equation (4):

Framing: Framing is a lossless process that divides continuous signals into overlapping, time-specific frames to reduce transition discontinuities. Using MFCC filtering, sound samples are represented as time functions with coefficients for frames centered at equally spaced intervals. Each speech segment—sounding or silent—is treated as a frame, with total frames equal to the sum of utterances and pauses. For example, the sentence “The joy of living is to love and respect” (5.871 s) includes utterances: “the” = 0.14 s, “joy” = 0.39 s, “of” = 0.08 s, “living” = 0.56 s, “is” = 0.10 s, “to” = 0.12 s, “love” = 0.47 s, “and” = 0.24 s, “respect” = 0.72 s, and pauses: 1.08, 0.26, 0.14, 0.33, 0.16, 1.06 s. The sounding (2.823 s) and silent durations (3.043 s) sum to the total (5.871 s), ensuring accurate, lossless framing.

4.1 Data

Broad representativeness requires a sufficiently large training dataset including utterances from male and female speakers. Since speech varies significantly across phonetic contexts, a comprehensive model requires at least 100,000 sentences. Manual recording is highly labor-intensive, involving content selection, phonetic variation coverage, participant recruitment, post-processing, and transcription. We utilized publicly available speech corpora, including the British National Corpus (BNC),²⁷ American National Corpus (ANC),²⁸ and Corpus of Contemporary American English (COCA),²⁹ selecting the Buckeye Speech Corpus³⁰ and EMU Speech Database³¹ for training. Buckeye comprises approximately 40 hours of conversational English (360,000 words or 24,000 sentences at 15 words/sentence). EMU contributes 30,000 sentences, yielding 54,000 total sentences. To meet the desired data volume, we applied augmentation techniques including pitch shifting (adjusting pitch without affecting duration to simulate various speaker profiles), time-stretching (modifying speech speed while preserving pitch for different speaking rates), volume alteration, background noise addition, and reverberation simulation to introduce acoustic variability. These methods increased the effective dataset to approximately 150,000 sentences. For storage, assuming mono audio at 16 kHz sampling rate and 16-bit resolution (32 KB/second), with 150,000 sentences averaging 5 seconds each as described in Equation 6:

The model is evaluated on all five corpora. Speech recognition tasks were implemented using Praat, ²¹ a phonetic analysis tool developed by Paul Boersma and David Weenink at the Amsterdam Institute of Phonetic Sciences, facilitating analysis, synthesis, and manipulation of speech signals for phonetics research.

4.1.1 Data partitioning and leakage prevention

To ensure independent training and evaluation, we partitioned all datasets using a 70-15-15 split for training, validation, and testing, respectively. For the British National Corpus (BNC) and American National Corpus (ANC), 70,000 sentences were allocated for training, 15,000 for validation, and 15,000 for testing. The Corpus of Contemporary American English (COCA) followed the same distribution with 70,000 training, 15,000 validation, and 15,000 testing sentences. For the Buckeye Speech Corpus, which contains approximately 24,000 sentences, 16,800 sentences (80% of the total) were used for training, 3,600 sentences (15%) for validation, and the remaining 3,600 sentences (15%) for testing. The EMU Speech Database, comprising 30,000 sentences, was split into 21,000 training sentences (70% of the total), 4,500 validation sentences (15%), and 4,500 testing sentences (15%). For our custom-built dataset of 2000 audio files, 1,400 files were used for training, 300 for validation, and 300 for testing.

Speaker-level partitioning was enforced to prevent data leakage, ensuring that no individual speaker’s voice appeared in both training and testing sets. For the Buckeye corpus, which contains approximately 40 hours of conversational speech from 40 speakers, we allocated 32 speakers exclusively for training, 4 speakers for validation, and 4 speakers for testing. Temporal windows were maintained as disjoint across all splits, meaning no overlapping time segments existed between training, validation, and testing data. All augmented data—including pitch shifting, time-stretching, volume alteration, background noise addition, and reverberation—were generated only after the initial partitioning was completed. This precaution prevented artificial inflation of performance metrics that could otherwise arise from augmented versions of training data leaking into validation or testing sets through correlated transformations. Cross-validation techniques were additionally employed during hyperparameter tuning to further assess model generalisation on unseen speech samples.

4.2 Windowing through Hidden Markov model

Each speech signal frame captures cepstral features characterizing the corresponding sound segment. Windowing derives grapheme-level representations for each phoneme within a frame. Hidden Markov Models (HMMs) generate sequences and patterns of hidden states based on observed acoustic features, facilitating phoneme-to-grapheme mapping. During preprocessing, speech signal $S$ is segmented into frames $\left\{f_n\right\}$ , with each frame $f_i$ subdivided into windows $\left\{w_n\right\}$ , where each window $w_i$ spans 0.015 s—optimal for preserving spectral information without temporal overlap or resolution loss. This is defined in Equations (7) and (8):

Each phoneme is modeled using a 3-state left-to-right HMM (standard in ASR literature), with states corresponding to: $q_1$ (onset), $q_2$ (steady-state), $q_3$ (offset). The transition matrix $A$ is initialized with zero probability for backward transitions ( $a_{\mathit{ij}}=0$ for $j<i$ ). This topology captures the sequential nature of phoneme articulation. For diphthongs and affricates, we tested 5-state models but found no significant WER improvement (0.3% reduction) for 67% increase in parameters, so the 3-state model was retained. Window formation follows Algorithm 1. Each acoustic feature extracted from a window maps to its corresponding language model component. Training the HMM classifier is crucial for accurate phoneme extraction. During training, known state sequences enable inference of unknown states. Training corpora include sound utterances for all syllable combinations with corresponding phonemic representations. Temporal overlap between consecutive windows or frames captures transitional features from previous states, improving current state learning. The overlap must balance containing at least one complete phoneme structure while avoiding excessive repetition. Based on empirical evaluation, overlap duration was set to 0.5 milliseconds between successive windows and frames.

Algorithm 1: HMM-based Windowing Process

Language features are extracted from each window as follows. For every window ( $w_i$ ), the corresponding phoneme is identified by matching acoustic features with pronunciation dictionary entries. If a unique phoneme is found, it is directly assigned and the process continues. When multiple phoneme candidates exist, probabilities are computed based on previously known state sequences, selecting the most probable phoneme. If no match is identified, an HMM infers the current state from prior known states. Finally, a dynamic text wrapping algorithm structures the phoneme combinations derived through HMM.

4.3 Backpropagation Neural Network (BPNN) in speech recognition

BPNN minimizes classification errors in speech-to-text conversion.³² Feature extraction techniques such as mel-frequency cepstral coefficients (MFCCs) transform raw audio into feature vector $\mathbf{x}$ , which BPNN processes to classify phonemes. Forward propagation computes neuron outputs in hidden and output layers:

The output layer generates predicted phoneme probability distributions, with error calculated using cross-entropy loss:

During backpropagation, error gradients are computed and propagated backward to adjust weights following gradient descent:

4.4 Algorithm: Adaptive Phoneme State Learning (APSL)

This algorithm enhances traditional BPNN-HMM speech recognition by introducing an adaptive mechanism that refines HMM state transitions based on BPNN confidence scores. The Adaptive Phoneme State Learning (APSL) algorithm combines a BPNN and HMM to dynamically learn phoneme transitions. Speech signals are segmented into overlapping 0.015 s windows, with cepstral features extracted using MFCCs. The Viterbi algorithm identifies the most probable phoneme state sequence by maximizing transition likelihoods given the trained HMM parameters,³³ while the BPNN classifies phonemes and updates weights via gradient descent using the cross-entropy loss function.

The confidence score in the APSL represents the reliability of phoneme classification using BPNN. This is defined as the posterior probability $P\left(p_j\right|x)$ , where $p_j$ is a phoneme and $x$ is the feature vector.³⁴ The confidence score helps in adaptive transition refinement, ensuring that phonemes with low classification certainty undergo additional training or an extended analysis.

If a phoneme’s confidence score is below a threshold $\theta$ , APSL dynamically modifies the HMM transition and emission probabilities. The updated emission probability is computed as:

Theoretical Derivation of Adaptive Probability Fusion

Let the posterior probability of the phoneme $p_j$ given acoustic feature $x$ from the BPNN be denoted as $P_{\mathit{NN}}\left(p_j|x\right)$ . Let the HMM emission probability be $P_{\mathit{HMM}}\left(x|q_i\right)$ . By Bayes' theorem, the hybrid posterior is Equation 17a, 17b and 17c:

However, direct multiplication assumes independence. To account for the fact that BPNN and HMM may have complementary errors, we employ a convex combination with uncertainty weighting:

where $\alpha \left(x\right)$ is not a fixed constant but a dynamic confidence-weighting function:

Here, ${\sigma }_{\mathit{NN}}^2$ and ${\sigma }_{\mathit{HMM}}^2$ are the empirical error variances estimated from validation data. This formulation is equivalent to Bayesian model averaging under the assumption of Gaussian-distributed estimation errors. When BPNN confidence is high ( ${\sigma }_{\mathit{NN}}^2\to 0$ ), $\alpha \left(x\right)\to 1$ ; when HMM is more reliable, $\alpha \left(x\right)\to 0$ .

To further improve recognition, APSL dynamically adjusts the window size for phonemes with low confidence scores:

The 5 ms increment was empirically optimized using grid search on a validation set (n=500 utterances), evaluating WER for Δt ∈ {2, 3, 5, 8, 10, 15} ms. Δt=5 ms yielded the optimal trade-off between phoneme boundary refinement (3.2% WER reduction) and computational overhead (≤8% latency increase). (i) Δt=2 ms: +0.8% WER reduction, +2% latency → suboptimal, (ii) Δt=5 ms: +3.2% WER reduction, +8% latency → optimal, (iii) Δt=10 ms: +3.5% WER reduction, +22% latency → diminishing returns. Thus, the 5 ms threshold corresponds to approximately one-third of a typical phoneme duration (15 ms), ensuring extension remains within the same phoneme rather than crossing boundaries.

The final phoneme sequence is determined by the Viterbi decoding process:

4.4.1 Optimal hyperparameter tuning using Bayesian optimization

Hyperparameter tuning is a critical step in machine learning for identifying the optimal set of hyperparameters to enhance model performance. Unlike model parameters learned during training, hyperparameters are predefined and govern the learning process, including the learning rate, number of hidden layers, batch size, and dropout rate. Selecting appropriate hyperparameters is essential for maximizing accuracy and minimizing errors. Bayesian Optimization is an efficient method for hyperparameter tuning, especially for complex models with expensive evaluation costs.³⁵ It constructs a probabilistic model of the objective function and uses an acquisition function to balance exploration and exploitation when selecting new hyperparameter configurations. Using Bayesian Optimization, optimal hyperparameters were determined for both the BPNN and APSL-BPNN-HMM speech recognition models. For the BPNN model, the optimal learning rate was 0.005, with three hidden layers of 256 neurons each, a batch size of 64, and 150 training epochs. The model employed the ReLU activation function with a dropout rate of 0.3, along with the Adam optimizer and cross-entropy loss function. For the APSL-BPNN-HMM model, the optimal learning rate was 0.003, with two hidden layers of 128 neurons each, a batch size of 64, and 200 epochs. The ReLU activation function with a dropout rate of 0.4 was used, while the confidence threshold ( $\theta$ ) was set to 0.75, the weighting factor ( $\alpha$ ) to 0.5, and the dynamic window adjustment size ( $\mathrm{\Delta }t$ ) to 15 ms. The Adam optimizer and cross-entropy loss function were also applied to ensure stable convergence and improved speech recognition accuracy.

5.1 Experimental set-up

The preprocessing phase is crucial for accurate and efficient speech recognition. To establish a robust dataset, 1000 audio files were manually created using mono channel setup with participants spanning ages 15-80, including fluent and non-fluent English speakers. All participants provided informed verbal consent following institutional ethical guidelines, as the study posed minimal risk and involved no sensitive personal data. Each participant used microphones and was presented with varying-length sentences. To introduce real-world variability, recordings were deliberately subjected to white and environmental noise. Noise was subsequently removed using high-pass and low-pass filters based on the Tunable Band Noise Reduction Technique (TBNRT) described in Section 3.2. The high-pass filter suppresses low-frequency noise:

The low-pass filter attenuates high-frequency noise:

where $f_c$ is the cutoff frequency based on detected noise profiles.

Recordings were conducted at four sampling frequencies: 18000, 32300, 44100, and 56000 Hz. Empirical results demonstrated superior performance at 44100 Hz, providing optimal balance between memory efficiency and audio clarity. Consequently, 44100 Hz was designated as the standardized sampling frequency. Additionally, 1000 audio files were sourced from online platforms featuring male and female speakers with diverse accents, including English and non-English speakers. The dataset covers multiple regions: i) Western European, ii) Eastern European, iii) Central Asia/Middle East/North African, iv) Sub-Saharan Africa, v) South Asia, vi) South East Asia, vii) CJK (Chinese, Japanese, Korean).³⁶ This expanded dataset totals 2000 files (3 seconds to 3 minutes duration, 0.9 GB storage), plus 24 GB from the corpus detailed in Section 4.1, posing substantial memory challenges during training. To address memory overhead, the APSL-BPNN-HMM framework employs an Adaptive Phoneme State Learning (APSL) mechanism for efficient parameter utilization. APSL introduces adaptive parameter sharing, dynamically assigning model parameters across layers to reduce redundancy through shared weight matrices between neighboring phoneme states. Consider a BPNN layer with $n$ input neurons, $m$ hidden neurons, and $p$ output neurons. Without APSL, total parameters are:

APSL integrates dynamic thresholding for parameter sharing control. During training, a similarity matrix $S_{\mathit{ij}}$ is computed between phoneme states $i$ and $j$ :

Validation of Memory-Accuracy Trade-off: To confirm that APSL's 32% memory reduction from 24.0 GB to 15.12 GB does not degrade accuracy, we conducted an ablation study comparing four configurations. The baseline HMM without APSL consumed 20.85 GB of memory and achieved 75.0% accuracy. APSL with full parameters where k equals m consumed 20.15 GB and achieved 95.8% accuracy, representing a 20.8% improvement over baseline. APSL with adaptive sharing where k equals 128 consumed 15.12 GB and achieved 96.0% accuracy, yielding a 21.0% improvement over baseline. APSL with aggressive sharing where k equals 64 consumed 11.80 GB but achieved only 92.3% accuracy, representing a 17.3% improvement over baseline. The adaptive sharing configuration with k equal to 128 actually improved accuracy slightly from 95.8% to 96.0% compared to the full parameter configuration. This occurs because parameter sharing acts as an implicit regularizer, reducing overfitting by limiting model complexity. The similarity threshold tau was set to 0.75, optimized on validation data to balance parameter sharing and state specificity. This reduction mitigates hardware constraints and accelerates model convergence by limiting parameter explosion, ensuring efficient resource utilization and scalability for large-scale speech recognition tasks.

Figure 2. A flowchart depicting the APSL mechanism from input features through feature extraction, similarity matrix computation, and threshold-based decision making to form shared or independent parameters, culminating in the final prediction.

Figure 3 demonstrates the optimization impact by comparing memory consumption across iterations for the baseline and proposed APSL-BPNN-HMM framework. The Baseline Model (skyblue) steadily increases memory usage, reaching approximately 20.85 GB at 5000 iterations, while APSL-BPNN-HMM (navy) maintains significantly lower usage, stabilizing around 15.15 GB. This reduction reflects APSL’s effectiveness in minimizing redundant parameter storage through dynamic thresholding and shared weight matrices. Adaptive parameter sharing reduces independent parameters, efficiently controlling model complexity without compromising accuracy. Consequently, APSL-BPNN-HMM achieves 32% memory reduction, accelerating convergence and enhancing scalability for large-scale tasks. An embedded subplot illustrates accuracy fluctuations across iterations and memory usage, showing APSL-BPNN-HMM and baseline HMM performance behavior. APSL-BPNN-HMM maintains higher accuracy while optimizing memory utilization. Yellow and blue markers indicate peak accuracies: APSL-BPNN-HMM (96%) and baseline HMM (75%), corresponding to their memory usage at that iteration. An enlarged contour plot emphasizes accuracy peaks for both models, with warmer colors indicating higher accuracy. APSL-BPNN-HMM achieves 96% peak accuracy at approximately 15.15 GB, while HMM reaches 75% at around 20.85 GB.

Figure 3. A line graph compares memory consumption across training iterations for the Baseline Model and APSL-BPNN-HMM, demonstrating reduced memory usage with adaptive parameter sharing, while an inset plot shows accuracy fluctuations for both models and an extended contour visualization highlights the memory–accuracy tradeoff at peak performance points.

Hardware and Software Environment

All experiments were conducted on a Windows-based computing workstation configured for speech recognition model training and evaluation. The system was equipped with an Intel Core i9 processor, providing high single-threaded performance suitable for sequential HMM processing tasks. System memory consisted of sufficient RAM for processing the largest training batches without disk swapping. For neural network acceleration, an NVIDIA GeForce RTX 4060 GPU was utilized exclusively for BPNN training, while HMM processing and feature extraction remained CPU-bound. Storage was provided by a high-speed NVMe solid-state drive, offering rapid data loading for the 24 GB dataset. The operating system was Windows (version unspecified but latest stable), chosen for compatibility with Praat and other phonetic analysis tools. Software dependencies included Python for model implementation, PyTorch for neural network operations, Praat for phonetic analysis and speech manipulation, and Librosa for audio feature extraction. Total training time for the APSL-BPNN-HMM model was approximately 72 hours to complete 200 training epochs, with the majority of computation time consumed by Viterbi decoding and adaptive windowing operations.

5.2 Metrics

5.2.1 Classification metrics: Recall, precision and F-score

To compute F-measure, recall and precision calculations are essential. Precision defines the ratio of correctly identified words to all recognized words ( Equation 25). For example, if ten speech features are identified as positive, precision measures transformation accuracy to correct textual information. Recall quantifies the percentage of specified keywords identified relative to all keywords that should have been identified (Equation 26). If 10 positive samples exist, recall measures classifier effectiveness in identifying correct features. F-score is the harmonic mean of recall and precision (Equation 27). These metrics utilize four classes: True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN),³⁷ defined as: i) True Positive (TP): Words present in audio are accurately retrieved as text (e.g., “living” in audio $\to$ “living” in text). ii) False Positive (FP): Words not in audio are retrieved as correct words (e.g., “Emanuel run the show” $\to$ “E manual run the show,” where “E Manual” doesn’t exist in audio). iii) False Negative (FN): Words in audio are not correctly retrieved (e.g., “geographical” and “transmission” $\to$ “geografical” and “transmition”). iv) True Negative (TN): Words absent in audio are not retrieved as text (e.g., “Hope” absent in audio and not transcribed).

Accuracy: Accuracy is calculated based on automatically trained words. For example, “joy” was not in the corpus but phonemes were automatically trained using HMM and retrieved. Measures considered: of total words in audio ( $A$ ), how many are exactly present ( $A^{+}$ ), how many were automatically trained ( $A^{\ast }$ ), and how many were not identified ( $A^{\prime }$ )? (Equation 28)

5.3 Error metrics

5.3.1 BLEU score calculation

The Bilingual Evaluation Understudy (BLEU) score evaluates machine-translated text quality against reference translations as shown in Equation 29:

5.3.2 WER score calculation

Word Error Rate (WER) evaluates Automatic Speech Recognition (ASR) systems and is given by the Equation 30: