GenDBN-Ensemble: A Hybrid Framework for Intrusion Detection in Network Traffic

Abstract

Distributed Denial of Service (DDoS) and Denial of Service (DoS) attacks remain a major challenge for network security due to their impact on service availability and their evolving attack patterns. Conventional intrusion detection systems (IDS), which always depend on static rules, have trouble adapting and are especially sensitive to class imbalance, which increases the rate of false alarms. This paper presents GenDBN-Ensemble, a hybrid intrusion detection framework that uses a dynamically weighted soft-voting ensemble to integrate an Adam-optimized Deep Belief Network (DBN), Support Vector Machine (SVM), and XGBoost classifier. The Synthetic Minority Over-Sampling Technique (SMOTE), which limits oversampling by predetermined class-specific upper bounds to lower the risk of overfitting, was applied only to the training data in order to address class imbalance. A recall-sensitive Genetic Algorithm (GA) fitness function is used to guide the selection of features, while DBN training is improved through a hybrid fine-tuning loss that combines cross-entropy and reconstruction errors and the ensemble adopts a diversity-aware weighting scheme. An 80-20 stratified train-test split was used to assess the framework on the CIDDS-001 dataset, and stratified five-fold cross-validation was used during GA optimization. Both binary and multi-class classification settings were used in the experiments. The model produced macro-averaged precision, recall, and F1-scores above 0.99 in the multi-class task and 99.98% detection accuracy with a low false alarm rate in the binary task. These findings show that the proposed framework demonstrates promising performance for the detection of DoS and DDoS attacks.