A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks

E. M. Dogo, O. J. Afolabi, N. I. Nwulu, B. Twala, C. O. Aigbavboa

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

220 Citations (Scopus)

Abstract

In this paper, we perform a comparative evaluation of seven most commonly used first-order stochastic gradient-based optimization techniques in a simple Convolutional Neural Network (ConvNet) architectural setup. The investigated techniques are the Stochastic Gradient Descent (SGD), with vanilla (vSGD), with momentum (SGDm), with momentum and nesterov (SGDm+n)), Root Mean Square Propagation (RMSProp), Adaptive Moment Estimation (Adam), Adaptive Gradient (AdaGrad), Adaptive Delta (AdaDelta), Adaptive moment estimation Extension based on infinity norm (Adamax) and Nesterov-accelerated Adaptive Moment Estimation (Nadam). We trained the model and evaluated the optimization techniques in terms of convergence speed, accuracy and loss function using three randomly selected publicly available image classification datasets. The overall experimental results obtained show Nadam achieved better performance across the three datasets in comparison to the other optimization techniques, while AdaDelta performed the worst.

Original languageEnglish
Title of host publicationProceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018
EditorsS. K. Niranjan, Veena Desai, Vijay S. Rajpurohit, M N Nadkatti
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages92-99
Number of pages8
ISBN (Electronic)9781538677094
DOIs
Publication statusPublished - Dec 2018
Event1st International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018 - Belagavi, India
Duration: 21 Dec 201823 Dec 2018

Publication series

NameProceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018

Conference

Conference1st International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018
Country/TerritoryIndia
CityBelagavi
Period21/12/1823/12/18

Keywords

  • Artificial Intelligence
  • deep learning
  • optimizers
  • performance measures
  • stochastic gradient descent

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Energy Engineering and Power Technology
  • Communication
  • Computational Mechanics
  • Electrical and Electronic Engineering
  • Mechanical Engineering
  • Control and Optimization

Fingerprint

Dive into the research topics of 'A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks'. Together they form a unique fingerprint.

Cite this