Hybrid deep learning for anti-money laundering: Unsupervised detection of emerging schemes via feature fusion and explainable artificial intelligence

Abstract

Traditional rule-based anti-money laundering (AML) transaction monitoring systems suffer from high false-positive rates and rigidity in detecting complex emerging risk. This limitation has prompted changes to the Financial Action Task Force (FATF) recommendation 16, mandating the use of advanced systems for detecting money laundering schemes in cross-border payments. This study developed a hybrid framework integrating VAE-learned behavioural latent factors, GNN-captured relational network signals, and rule-based heuristics for enhanced anomaly detection. The model was evaluated on 54,258 real-world cross-border transaction records from an East African commercial bank.The One-Class SVM, optimized via a rigorous grid search proved superior compared to Isolation Forest and Local Outlier Factor benchmark, achieving a precision of 99.63% in the top 5% of prioritized alerts. Independent validation by a Kenyan financial institution confirms a batch processing speed of 1000 transactions per second on standard computer hardware (Intel Core i7, 16GB RAM) and efficient high-priority alert triage, key requirements for deployment in financial institutions. Shapley additive explanations analysis further provided the interpretability of the feature contribution to the model performance. These results demonstrated that integration of rule-based features with deep-learning embeddings improves compliance work efficiency and proven pathway for resource-constrained financial institutions to comply with FATF regulatory demands upcoming in 2030.