Impact of Optimization Algorithms and Regularization Techniques on Neural Network Performance for Binary Classification

Abstract

Optimization and regularization are two critical components influencing the convergence behavior and generalization ability of neural networks. This paper presents a comprehensive comparative evaluation of how different optimization algorithms-Adam, RMSprop, Gradient Descent (GD), and Gradient Descent with Momentum- interact with L1 and L2 regularization in a binary classification setting. Using the Fetal Health dataset from Kaggle, the models were trained over 25 epochs with a batch size of 32, and evaluated through accuracy, precision, recall, and Fl-score metrics. The results prove that the Adam optimizer with L1 regularization achieved the highest performance, reaching 94% accuracy, 0.94 weighted precision, 0.94 recall, and 0.94 Fl-score, closely followed by Adam with L2 and RMSprop with L1, both supporting accuracies above 93%. Conversely, pure gradient descent without momentum underperformed, achieving only 58% accuracy with L1 and 77% with L2, highlighting its inefficiency in handling non-convex error landscapes. Incorporating momentum significantly improved GD's stability and performance, achieving accuracies around 89-90%. These results emphasize the constructive collaboration between adaptive optimizers and proper regularization in improving model robustness and convergence speed. Future research will extend these findings to hybrid optimizers, adaptive regularization methods, and transfer learning-based binary classifiers.