Fractional-order Gradient Descent for Enhancing Deep Learning Optimization

Abstract

This study introduces Fractional-Order Gradient Descent (FGD) as an advanced optimization technique to enhance deep learning efficiency. Traditional gradient-based optimizers, such as Stochastic Gradient Descent (SGD) and Adam, often struggle with slow convergence, hyperparameter sensitivity, and entrapment in suboptimal local minima. To address these issues, FGD integrates fractional-order derivatives, leveraging memory effects and historical dependencies to improve optimization dynamics. Specifically, this study employs the Caputo fractional derivative to modify gradient updates, facilitating a balance between local exploration and global convergence. Experimental evaluations on benchmark datasets—including MNIST, CIFAR-10, and ImageNet—demonstrate that FGD outperforms conventional optimizers in terms of convergence speed, loss reduction, and robustness against noisy gradients. The proposed optimization method demonstrates faster convergence compared to the baseline methods, particularly on the more complex datasets. For MNIST, the convergence speed is similar across all methods due to the simplicity of the dataset, but the method reaches a stable loss value in fewer epochs than both Adam and RMSprop. The improved convergence speed of the method could be advantageous for environments with limited computational resources, where faster convergence could lead to cost savings and more effective utilization of hardware. Despite computational overhead and parameter tuning complexities, FGD emerges as a promising alternative for deep learning optimization, offering efficiency gains in training deep neural networks.