A Hybrid SMOTE-CTGAN and VAE-LSTM Framework for Interpretable Intrusion Detection in Imbalanced Network Traffic

Abstract

The increasing sophistication of cyber threats and severe class imbalance in network traffic continue to challenge traditional intrusion detection systems. This study proposes a hybrid framework that integrates SMOTE and CTGAN for minority-class augmentation, a Bidirectional Long Short-Term Memory (Bi-LSTM) network for supervised traffic classification, and a benign-trained Variational Autoencoder (VAE) for validating low-confidence predictions. The framework was evaluated on the CSE-CIC-IDS2018 dataset. On a final holdout test partition of 2,759,227 network flows, it achieved a binary accuracy of 98.51%, an F1-score of 92.59%, and a ROC-AUC of 0.9935. In the 15-class evaluation, it achieved 98.50% accuracy and a weighted F1-score of 98.42%, while stratified 10-fold cross-validation yielded a mean accuracy of 98.79%. The VAE was activated for 519,389 low-confidence flows, representing 18.82% of the complete holdout test partition, and primarily reduced false-positive predictions, although this improvement was accompanied by a measurable reduction in attack recall. SHAP analysis was applied to the supervised Bi-LSTM component to provide feature-level interpretability. The framework also achieved a mean inference latency of 0.5195 ms per network flow. These findings demonstrate strong aggregate detection performance, stable generalisation, effective false-alarm reduction, and low inference latency, while highlighting continuing challenges in rare-class and open-set detection.