Malware Detection Using RNA Encoding and Convolutional Neural Networks on the Malicious Network Dataset

1. Introduction

The exponential growth of networked systems proliferation has led to an equivalent increase in advanced malware attacks. Intrusion detection systems (IDS) continue to play a central role in the protection of digital infrastructure; however, modern practices are characterized by severe limitations.¹ Signature-based IDS rely on a set of pre-defined rules or patterns, and these systems are effective only against previously identified threats. Anomaly-based IDS attempt to detect deviations from normal behavior, and has the ability to detect zero-day attacks, but often generate a high false positive. To address these limitations, a hybrid architecture has arisen, combining the benefits of each of the system.^2,3

There have been recent studies into how deep-learning architectures, especially convolutional neural networks (CNNs) and recurrent neural networks (RNNs), can be applied to the problems of IDS, with promising results. A novel intrusion detection method based on learning framework is proposed,⁴ where the proposed method is done by using dual parallel CNN pipelines to independently address the network and radar features. A new intrusion detection model is suggested by,⁵ where this model combines CNN and Random Forest. The CNN is utilized to extract the feature, and the Random Forest is used for classification. An IDS is proposed by combining an innovative hybrid Autoencoder with an enhanced LSTM-CNN architecture,⁶ where the proposed method can enhance the detection capabilities more quickly and efficiently. Kaissar et al.⁷ investigates the optimization of hyperparameters in CNN to enhance the NIDS performance, where Grid Search, Genetic Algorithm, Particle Swarm Optimization, and Grey Wolf Optimization algorithms are used for this purpose. Alrayes et al.⁸ suggested a novel IDS by combining channel attention and CNN, where the suggested method has exceptional accuracy when applied it to NSL-KDD dataset. A new IDS model is built based on CNN and knowledge distillation,⁹ this model using two-dimensional Fourier transform for converting the grayscale images to the frequency domain, and this led to enhanced the similarity between neighboring pixels to address data effectively. Ban et al.¹⁰ suggested an enhanced deep-learning model for IDS in IoT environment, where the suggested model is depending on CNN as the backbone network in the constructed model. A hybrid deep learning IDS is proposed by¹¹ based on CNN and bidirectional long short-term memory neural networks, where the proposed system is enhanced the model’s ability to detect patterns in both minority and majority classes. Altunay and Albayrak¹² developed IDS in the IIoT networks, where the suggest system is done by using three different deep learning architectures, which are CNN, Long Short-Term Memory (LSTM), and the combination of these two methods.

Parallel Encodings Biologically inspired encodings, like mappings to DNA and RNA sequences, have been proposed to convert heterogeneous data to symbolic strings.^13,14 Nevertheless, there is limited evidence in the extant literature of the integration of RNA encodings with CNN classifiers in the entire range of detection paradigms: signature, anomaly, and hybrid. The current paper seals this gap by suggesting a CNN-based model which incorporates these complementary detection schemes. Despite the good outcomes of the current methods, a number of shortcomings still exist. Most CNN-based approaches use traditional data representations, which might not best represent intricate feature interactions, leading to worse performance in adverse conditions. Moreover, certain methods have higher computational costs and reduced resilience to a variety of data or when used on noisy data. These drawbacks indicate why encoding methods should be more effective and articulate. Here the proposed approach refers to encoding based on RNA to increase the feature representation to allow the model to learn more discriminative features to enhance the performance of the model in comparison with the current methods.

2. Methodology

The proposed malware detection system is built by combining RNA-inspired encoding of the network traffic characteristics and the convolutional neural network (CNN) classification. Unlike traditional intrusion detection system models that treat signature-based and anomaly-based detection methodologies as dissimilar entities, the current system integrates both of them in a single deep learning pipeline. In this pipeline, CNN models that are trained on sequences coded using the RNA-inspired methodology concurrently address signature-based, anomaly-based, and hybrid detection. The steps of the proposed system are shown in Figure 1.

Figure 1. Flow chart of the suggested malware detection system with RNA encoding and CNN classification pipeline.

2.3 CNN architecture

The coded messages are fed into a Convolutional Neural Network (CNN) that picks up discriminative features at a variety of levels of abstraction as follow:

• ○ Embedding: The codons are first mapped to dense vectors of dimension, d = 32. This embedding is learnt alongside the classifier, thus, encoding similarities between codons.

• ○ Convolutional: A number of one-dimensional convolutional layers are used, the size of which varies between 5 and 7. These filters identify a local pattern in the codon sequences e.g., repeated sequences which can be an indication of malicious activity. An example would be to have a convolution filter that is trained to identify the codon sequence of known back door ports.

• ○ Pooling: The feature maps are down sampled through max-pooling, and the most conspicuous features are retained, with a lower computational cost.

• ○ Global Average Pooling: In order to generalize over a wide range of lengths of variable sequences and reduce overfitting, global average pooling is done to aggregate the feature maps into fixed-size vectors.

• ○ Dense Layers: The learned features are combined in fully connected layers (64 units) that use ReLU activation. To inhibit overfitting, dropout regularization is used, with a temporary activation of neurons in the course of training, i.e. p = 0.5.

• ○ Output Layer: One sigmoid neuron generates a probability score which can mark a sample to be benign or malicious.

• ○ The CNN architecture is applied for three different methods, and these methods are shown in Figure 2.

Figure 2. The three CNN models (Signature, Anomaly, and Hybrid).

In Signature-CNN, the Signature- CNN replaces traditional rule-based signature matching with a convolutional neural network, which is trained on representations of known patterns of malicious activity encoded by RNA. Instead of searching manually through the collection of byte sequences or hash values, the network is trained to identify codon-level motifs which are indicative of malicious flows. While the Anomaly- CNN is trained to identify deviation with the normal network behavior using sequences encoded by RNA. It is not based on predefined attack patterns as compared to the signature model. Finally, the Hybrid-CNN combines the two methods in two-staged pipeline, allowing the Signature-CNN to combine the precision of Anomaly-CNN with the generalization capability. The comparison between the three CNN models is clarified in Table 3.

Table 3. Comparison between different CNN models.

Model	Input	Objective	Strengths	Weaknesses
Signature-CNN	RNA-coded sequences of known malicious and benign flows	Detect known attack patterns using learned codon-level motifs	High precision and low FAR	Cannot detect zero-day threats
Anomaly-CNN	RNA-coded mixture of benign and malicious flows	Identify deviations from normal codon distributions	Detects novel and polymorphic malware	Higher false-positive rate
Hybrid-CNN	Two-stage input: Signature-filtered and residual flows	Combine the benefits of both precision and generalization	Best overall accuracy and balance	Slightly higher computational cost

Where the sequence handling all the RNA sequences were truncated or zero padded to a constant length of 2,048 codons to make the input equal. Also, the architectural consistency for the CNN models of the three models have the same architecture (three Conv1D layers with 64-128-256 filters, kernel = 5, ReLU activation, and dropout = 0.5). It is only in training objectives that there is a difference. Finally, the dataset was stratified 70/30 and sampled to maintain the malicious/benign ratio (50/50). In particular, the amount of 10000 malicious and 10000 benign samples were maintained to be equal when training and testing the experimental dataset. These explanations guarantee the maximum reproducibility of the experiment. The CNN model is optimized through Adam optimization algorithm of learning rate 0.001. The training is undertaken through 50 epochs having a batch size that is 32. In a bid to reduce overfitting, dropout regularization, with a dropout rate of 0.5, is used, and early stopping is utilized, on the basis of validation loss. The model has one output neuron (sigmoid) for binary classification, and optimization is done using the binary cross-entropy loss function. During training, the Adam optimizer with a learning rate of 0.001 is used.

Deep learning model development was done with the help of Python 3.11, TensorFlow 2.15, and Keras. Other libraries used were NumPy for numerical computation, Pandas for data preprocessing and Scikit-learn for evaluation metrics and splitting the data. All experiments were performed on an NVIDIA RTX-4090 with 24GB of VRAM.

3. Results and discussion

The performance of the proposed CNN-based malware detection models is evaluated based on several standard classification metrics were employed, where these metrics are defined and calculated as follow:

The identified enhancement of the performance with the help of RNA encoding can be explained by the capacity of the solution to increase the feature representation in the CNN framework. RNA encoding converts the input data into structured and biologically inspired encoding, which adds further diversity and non-linearity to the feature space. This transformation allows the network to pick up more complex and discriminative patterns that might not be available with traditional encoding methods. Additionally, RNA encoding also helps in reducing noise and enhancing generalization as it focuses on meaningful relationships in the data. Due to this, the CNN can develop more resilient features resulting in higher classification accuracy and system performance. The Malicious Network Dataset is used for evaluation the proposed method, where each method (signature, anomaly, or hybrid) is starting by preprocessing the used dataset by removing IP fields, then RNA encoding is applied to all remaining features. Then divided the dataset into training and testing, the training is equal to 70%, while the testing used the rest 30% from the whole dataset. The achieved results for the first method (signature-CNN) are shown in Table 4. The Signature-CNN was very accurate and reported a low false-positive rate, thus, justifying its accuracy in identifying known malware.

While the obtained results based on the second method (anomaly-CNN) are shown in Table 5. The Anomaly-CNN was able to achieve higher rate of detection, which revealed that zero-day threats can be spotted with a slight increase in false positives .

On other hand, when utilized the third method (hybrid-CNN), the achieved results are shown in Table 6. Hybrid-CNN delivered the best trade-off, achieving the best detection rate and F1 score whilst reducing false positives at the same time.

Finally, the comparison between the performance of all models (signature, anomaly, and hybrid) are shown in Table 7 and Figure 3.

Figure 3. Comparison of Signature-CNN, Anomaly-CNN and Hybrid-CNN on the malicious network dataset.

As shown in Table 7, the Hybrid-CNN achieved the best overall results among the evaluated methods, where the obtained accuracy, detection rate, precision, F1 score, and FPR are equal to 95%, 95%, 94%, 94.5%, and 3% respectively. The Hybrid-CNN may be explained by the possibility of making decisions in two stages. The first stage filters familiar malicious patterns and the latter extrapolates to unknown codon patterns. This is a combination that reduces the antagonism of sensitivity and specificity. As compared to the Anomaly-CNN, however, it has a slightly higher recall with more false positives since it does not identify exact patterns but only the deviations. The Signature-CNN is also accurate to known dangers but does not have generalization and this is the reason behind its slightly lower recall. In order to get more in-depth knowledge of classification behavior of proposed models, confusion matrices for Signature-CNN, Anomaly-CNN, and Hybrid-CNN are shown in Figure 4. These matrices show the distribution of true positive (TP), true negative (TN), false positive (FP) and false negative (FN) cases and provide a clearer picture of the reported evaluation metrics.

Figure 4. Confusion matrices of the proposed Signature-CNN, Anomaly-CNN, and Hybrid-CNN models on the Malicious Network Dataset.

The training and validation accuracy/loss curve of the proposed Hybrid-CNN model are shown in Figure 5. The curves show that the algorithm converged in a stable manner during training and that the different measures taken to prevent overfitting (dropout regularization, early stopping) did not seem to sacrifice generalization ability.

Figure 5. The training and validation accuracy and loss curves of the proposed Hybrid-CNN model. Curves show stable convergence during training, and that dropout regularization and early stopping proved to be effective in decreasing overfitting while keeping good generalization performance.

The proposed method achieved results are compared with classical machine learning models (Random Forest, and XGBoost) and deep models (RNN, CNN-BiLSTM and AE-LSTM) and this comparison is shown in Table 8.

As shown in Table 8, The Hybrid-CNN achieved better results than the traditional and deep learning methods. Where the proposed method achieved the highest accuracy, detection rate, precision, and F1-score, where these results are equal to 95%, 95%, 94%, and 94.5% respectively. Also, the obtained FPR results are the lowest and equal to 3%. This has been enhanced by the fact that it has RNA encoding that maintains semantical links between traffic features and increases the pattern recognition capability of the CNN.

4. Conclusion

This work proposed an integrated malware detector model based on CNN, which was built on RNA encoding and implemented on Malicious Network Dataset. Restructuring signature, anomaly, and hybrid detection as CNN-based paradigms, the system achieves strong performance across all detection modes. The Hybrid-CNN achieved the best results, having 95% of detection, and the same time, minimized false-positive risks. Future directions will be to extrapolate the proposed technique to bigger and more heterogeneous datasets to further test the generalization capability of the technique. Moreover, hybrid deep learning models, including CNN architecture and the use of transformer-based methods, will be considered to improve feature learning. The other valuable direction is the optimization of the model to real-time applications and minimization of the complexity of the computation. Also, the exploration of other bio-inspired encoding methods can offer further enhancements in feature representation and efficiency of the model.