A Survey of Advanced Gradient Methods in Machine Learning

Abstract

Gradient-based optimization methods are integral to training machine learning models across various tasks, serving as the primary tools for minimizing loss functions and improving performance. Classical methods, such as Batch Gradient Descent (BGD) and Stochastic Gradient Descent (SGD), have been widely used for their simplicity and effectiveness. However, these approaches often face challenges, including slow convergence rates, hyperparameter sensitivity, and difficulties in non-convex optimization. Recent advancements in adaptive methods, such as AdaGrad, RMSProp, and Adam, have addressed some of these issues by dynamically adjusting learning rates. Momentum-based techniques like Nesterov Accelerated Gradient (NAG) further enhance convergence speed and stability in high-dimensional spaces. This paper provides a comprehensive overview of these optimization techniques, highlighting their respective strengths and limitations while also exploring advanced methods like MADGRAD, which integrates momentum and adaptive learning. The discussion emphasizes the need for balancing computational efficiency, convergence speed, and scalability, particularly in large-scale deep learning applications.