An Optimized Deep Learning Approach for Multiclass Anomaly Detection

Abstract

The increasing scale and imbalance of modern network traffic pose significant challenges for multi-class intrusion detection systems (IDSs), particularly in identifying rare attack types. Traditional intrusion detection approaches based on supervised classification or unsupervised anomaly detection often suffer from limited generalization under severe class imbalance, high-dimensional feature spaces, and noisy traffic, resulting in poor detection of minority attack classes. To address these limitations, this study presents a hybrid intrusion detection framework that integrates unsupervised feature learning, anomaly scoring, and supervised classification within a unified pipeline. A denoising autoencoder trained exclusively on normal traffic is employed to learn compact and noise-resistant feature representations, while an isolation forest independently generates statistical anomaly scores. These complementary features are then fused and classified using a Light Gradient Boosting Machine (LightGBM). The main contribution of this work lies in the effective integration of these components, combined with a balanced training strategy based on the Synthetic Minority Over-sampling Technique with Edited Nearest Neighbors (SMOTE-ENN), as well as robust validation procedures. The framework is evaluated on the Network Security Laboratory Knowledge Discovery and Data Mining dataset (NSL-KDD) and the UNSW-NB15 intrusion detection dataset using stratified cross-validation and multiple independent runs. Experimental results demonstrate consistently high classification accuracy (~99%) and strong macro-F1 performance (>97%) across all attack categories on both NSL-KDD and UNSW-NB15 datasets. The framework achieves exceptional detection of rare classes (R2L: 99% F1, U2R: 100% F1), significantly outperforming prior approaches (AE-SAC: 83.97% F1, RL-NIDS: poor U2R recall), while maintaining low inference latency (~2–3 ms per sample, 415 samples/second) suitable for real-time network security deployment.