Survey of Gradient Descent Variants and Evaluation Criteria

Abstract

Gradient descent is a fundamental optimization algorithm widely used in artificial intelligence to minimize the loss function and find the optimal parameters of a model, so optimize the learning process. It is one of the most important algorithms for training various types of models, such as neural networks, support vector machines, linear regression, and so on. It arouses the keen interest and enthusiasm of a very large group of researchers. Indeed, there has been a surge in research on gradient descent variants and their applications in various fields to enhance the efficiency, convergence speed and rate, accuracy of the optimization process, and generalization performance of the algorithm over the years. The large number of gradient descent variants, their variety and diversity lead to confusion and ambiguity. In addition, these variants are scattered and dispersed in the mass of scientific research, which is a source of undecidability in choosing an appropriate variant. In order to overcome this issue and address this challenge, we provide a comprehensive survey of gradient descent variants and evaluation criteria to support researchers, practitioners, readers, and others to make informed decisions when choosing an optimization algorithm for their artificial intelligence project, leading to more efficient and effective optimization processes.