The exponential growth of networked systems proliferation has led to an equivalent increase in advanced malware attacks. Intrusion detection systems (IDS) continue to play a central role in the protection of digital infrastructure; however, modern practices are characterized by severe limitations. ¹ Signature-based IDS rely on a set of pre-defined rules or patterns, and these systems are effective only against previously identified threats. Anomaly-based IDS attempt to detect deviations from normal behavior, and has the ability to detect zero-day attacks, but often generate a high false positive. To address these limitations, a hybrid architecture has arisen, combining the benefits of each of the system. ^{2, 3}

There have been recent studies into how deep-learning architectures, especially convolutional neural networks (CNNs) and recurrent neural networks (RNNs), can be applied to the problems of IDS, with promising results. A novel intrusion detection method based on learning framework is proposed, ⁴ where the proposed method is done by using dual parallel CNN pipelines to independently address the network and radar features. A new intrusion detection model is suggested by, ⁵ where this model combines CNN and Random Forest. The CNN is utilized to extract the feature, and the Random Forest is used for classification. An IDS is proposed by combining an innovative hybrid Autoencoder with an enhanced LSTM-CNN architecture, ⁶ where the proposed method can enhance the detection capabilities more quickly and efficiently. Kaissar et al. ⁷ investigates the optimization of hyperparameters in CNN to enhance the NIDS performance, where Grid Search, Genetic Algorithm, Particle Swarm Optimization, and Grey Wolf Optimization algorithms are used for this purpose. Alrayes et al. ⁸ suggested a novel IDS by combining channel attention and CNN, where the suggested method has exceptional accuracy when applied it to NSL-KDD dataset. A new IDS model is built based on CNN and knowledge distillation, ⁹ this model using two-dimensional Fourier transform for converting the grayscale images to the frequency domain, and this led to enhanced the similarity between neighboring pixels to address data effectively. Ban et al. ¹⁰ suggested an enhanced deep-learning model for IDS in IoT environment, where the suggested model is depending on CNN as the backbone network in the constructed model. A hybrid deep learning IDS is proposed by ¹¹ based on CNN and bidirectional long short-term memory neural networks, where the proposed system is enhanced the model’s ability to detect patterns in both minority and majority classes. Altunay and Albayrak ¹² developed IDS in the IIoT networks, where the suggest system is done by using three different deep learning architectures, which are CNN, Long Short-Term Memory (LSTM), and the combination of these two methods.

Parallel Encodings Biologically inspired encodings, like mappings to DNA and RNA sequences, have been proposed to convert heterogeneous data to symbolic strings. ^{13, 14} Nevertheless, there is limited evidence in the extant literature of the integration of RNA encodings with CNN classifiers in the entire range of detection paradigms: signature, anomaly, and hybrid. The current paper seals this gap by suggesting a CNN-based model which incorporates these complementary detection schemes.

The proposed malware detection system is built by combining of RNA-inspired encoding of the network traffic characteristics and the convolutional neural network (CNN) classification. Unlike traditional intrusion detection system models that treat signature-based and anomaly-based detection methodologies as dissimilar entities, the current system integrates both of them in a single deep learning pipeline. In this pipeline, CNN models that are trained on sequences coded using the RNA-inspired methodology concurrently address signature-based, anomaly-based, and hybrid detection. The steps of the proposed system are shown in Figure 1.

The performance of the proposed CNN-based malware detection models is evaluated based on several standard classification metrics were employed, where these metrics are defined and calculated as follow: ○

Accuracy: Is the ratio of correctly labeled flows to the total number of flows, and it calculate based on the following equation: Accuracy = TP + TN TP + TN + FP + FN

The Malicious Network Dataset is used for evaluation the proposed method, where each method (signature, anomaly, or hybrid) is starting by preprocessing the used dataset by removing IP fields, then RNA encoding is applied to all remaining features. Then divided the dataset into training and testing, the training is equal to 70%, while the testing used the rest 30% from the whole dataset. The achieved results for the first method (signature-CNN) are shown in Table 4. The Signature-CNN was very accurate and reported a low false-positive rate, thus, justifying its accuracy in identifying known malware.

While the obtained results based on the second method (anomaly-CNN) are shown in Table 5. The Anomaly-CNN was able to achieve higher rate of detection, which revealed that zero-day threats can be spotted with a slight increase in false positives .

On other hand, when utilized the third method (hybrid-CNN), the achieved results are shown in Table 6. Hybrid-CNN delivered the best trade-off, achieving the best detection rate and F1 score whilst reducing false positives at the same time.

Finally, the comparison between the performance of all models (signature, anomaly, and hybrid) are shown in Table 7 and Figure 3.

As shown in Table 7, the achieved results based on hybrid-CNN are achieved the best results over the two others methods, where the obtained accuracy, detection rate, precision, F1 score, and FPR are equal to 95%, 95%, 94%, 94.5%, and 3% respectively. The Hybrid-CNN may be explained by the possibility of making decisions in two stages. The first stage filters familiar malicious patterns and the latter extrapolates to unknown codon patterns. This is a combination that reduces the antagonism of sensitivity and specificity. As compared to the Anomaly- CNN, however, it has a slightly higher recall with more false positives since it does not identify exact patterns but only the deviations. The Signature-CNN is also accurate to known dangers but does not have generalization and this is the reason behind its slightly lower recall.

The proposed method achieved results are compared with classical machine learning models (Random Forest, and XGBoost) and deep models (RNN, CNN-BiLSTM and AE-LSTM) and this comparison is shown in Table 8.

As shown in Table 8, The Hybrid-CNN achieved better results than the traditional and deep learning methods. Where the proposed method achieved the highest accuracy, detection rate, precision, and F1-score, where these results are equal to 95%, 95%, 94%, and 94.5% respectively. Also, the obtained FPR results are the lowest and equal to 3%. This has been enhanced by the fact that it has RNA encoding that maintains semantical links between traffic features and increases the pattern recognition capability of the CNN.

This work proposed an integrated malware detector model based on CNN, which was built on RNA encoding and implemented on Malicious Network Dataset. Restructuring signature, anomaly, and hybrid detection as CNN-based paradigms, the system achieves strong performance across all detection modes. The Hybrid-CNN achieved the best results, having 95% of detection, and the same time, minimized false-positive risks. For future work, the optimization of RNA encoding to stream traffic to allow real-time deployment, lightweight inference latency exploring lightweight architecture like Transformers and Graph Neural Networks, and scaling experiments to larger and more diverse network datasets. Moreover, it will be investigated in association with federated learning models in order to improve data privacy and improve the model generalization across network settings.