A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks

E. M. Dogo; O. J. Afolabi; N. I. Nwulu; B. Twala; C. O. Aigbavboa

doi:10.1109/CTEMS.2018.8769211

A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks

E. M. Dogo, O. J. Afolabi, N. I. Nwulu, B. Twala, C. O. Aigbavboa

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

279 Citations (Scopus)

Abstract

In this paper, we perform a comparative evaluation of seven most commonly used first-order stochastic gradient-based optimization techniques in a simple Convolutional Neural Network (ConvNet) architectural setup. The investigated techniques are the Stochastic Gradient Descent (SGD), with vanilla (vSGD), with momentum (SGDm), with momentum and nesterov (SGDm+n)), Root Mean Square Propagation (RMSProp), Adaptive Moment Estimation (Adam), Adaptive Gradient (AdaGrad), Adaptive Delta (AdaDelta), Adaptive moment estimation Extension based on infinity norm (Adamax) and Nesterov-accelerated Adaptive Moment Estimation (Nadam). We trained the model and evaluated the optimization techniques in terms of convergence speed, accuracy and loss function using three randomly selected publicly available image classification datasets. The overall experimental results obtained show Nadam achieved better performance across the three datasets in comparison to the other optimization techniques, while AdaDelta performed the worst.

Original language	English
Title of host publication	Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018
Editors	S. K. Niranjan, Veena Desai, Vijay S. Rajpurohit, M N Nadkatti
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	92-99
Number of pages	8
ISBN (Electronic)	9781538677094
DOIs	https://doi.org/10.1109/CTEMS.2018.8769211
Publication status	Published - Dec 2018
Event	1st International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018 - Belagavi, India Duration: 21 Dec 2018 → 23 Dec 2018

Publication series

Name	Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018

Conference

Conference	1st International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018
Country/Territory	India
City	Belagavi
Period	21/12/18 → 23/12/18

Keywords

Artificial Intelligence
deep learning
optimizers
performance measures
stochastic gradient descent

ASJC Scopus subject areas

Computer Networks and Communications
Computer Science Applications
Energy Engineering and Power Technology
Communication
Computational Mechanics
Electrical and Electronic Engineering
Mechanical Engineering
Control and Optimization

Access to Document

10.1109/CTEMS.2018.8769211

Cite this

Dogo, E. M., Afolabi, O. J., Nwulu, N. I., Twala, B., & Aigbavboa, C. O. (2018). A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks. In S. K. Niranjan, V. Desai, V. S. Rajpurohit, & M. N. Nadkatti (Eds.), Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018 (pp. 92-99). Article 8769211 (Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CTEMS.2018.8769211

Dogo, E. M. ; Afolabi, O. J. ; Nwulu, N. I. et al. / A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks. Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018. editor / S. K. Niranjan ; Veena Desai ; Vijay S. Rajpurohit ; M N Nadkatti. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 92-99 (Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018).

@inproceedings{4094eeee761b45dd913b1ded049af55b,

title = "A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks",

abstract = "In this paper, we perform a comparative evaluation of seven most commonly used first-order stochastic gradient-based optimization techniques in a simple Convolutional Neural Network (ConvNet) architectural setup. The investigated techniques are the Stochastic Gradient Descent (SGD), with vanilla (vSGD), with momentum (SGDm), with momentum and nesterov (SGDm+n)), Root Mean Square Propagation (RMSProp), Adaptive Moment Estimation (Adam), Adaptive Gradient (AdaGrad), Adaptive Delta (AdaDelta), Adaptive moment estimation Extension based on infinity norm (Adamax) and Nesterov-accelerated Adaptive Moment Estimation (Nadam). We trained the model and evaluated the optimization techniques in terms of convergence speed, accuracy and loss function using three randomly selected publicly available image classification datasets. The overall experimental results obtained show Nadam achieved better performance across the three datasets in comparison to the other optimization techniques, while AdaDelta performed the worst.",

keywords = "Artificial Intelligence, deep learning, optimizers, performance measures, stochastic gradient descent",

author = "Dogo, {E. M.} and Afolabi, {O. J.} and Nwulu, {N. I.} and B. Twala and Aigbavboa, {C. O.}",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 1st International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018 ; Conference date: 21-12-2018 Through 23-12-2018",

year = "2018",

month = dec,

doi = "10.1109/CTEMS.2018.8769211",

language = "English",

series = "Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "92--99",

editor = "Niranjan, {S. K.} and Veena Desai and Rajpurohit, {Vijay S.} and Nadkatti, {M N}",

booktitle = "Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018",

address = "United States",

}

Dogo, EM, Afolabi, OJ, Nwulu, NI, Twala, B & Aigbavboa, CO 2018, A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks. in SK Niranjan, V Desai, VS Rajpurohit & MN Nadkatti (eds), Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018., 8769211, Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018, Institute of Electrical and Electronics Engineers Inc., pp. 92-99, 1st International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018, Belagavi, India, 21/12/18. https://doi.org/10.1109/CTEMS.2018.8769211

A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks. / Dogo, E. M.; Afolabi, O. J.; Nwulu, N. I. et al.
Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018. ed. / S. K. Niranjan; Veena Desai; Vijay S. Rajpurohit; M N Nadkatti. Institute of Electrical and Electronics Engineers Inc., 2018. p. 92-99 8769211 (Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks

AU - Dogo, E. M.

AU - Afolabi, O. J.

AU - Nwulu, N. I.

AU - Twala, B.

AU - Aigbavboa, C. O.

PY - 2018/12

Y1 - 2018/12

N2 - In this paper, we perform a comparative evaluation of seven most commonly used first-order stochastic gradient-based optimization techniques in a simple Convolutional Neural Network (ConvNet) architectural setup. The investigated techniques are the Stochastic Gradient Descent (SGD), with vanilla (vSGD), with momentum (SGDm), with momentum and nesterov (SGDm+n)), Root Mean Square Propagation (RMSProp), Adaptive Moment Estimation (Adam), Adaptive Gradient (AdaGrad), Adaptive Delta (AdaDelta), Adaptive moment estimation Extension based on infinity norm (Adamax) and Nesterov-accelerated Adaptive Moment Estimation (Nadam). We trained the model and evaluated the optimization techniques in terms of convergence speed, accuracy and loss function using three randomly selected publicly available image classification datasets. The overall experimental results obtained show Nadam achieved better performance across the three datasets in comparison to the other optimization techniques, while AdaDelta performed the worst.

AB - In this paper, we perform a comparative evaluation of seven most commonly used first-order stochastic gradient-based optimization techniques in a simple Convolutional Neural Network (ConvNet) architectural setup. The investigated techniques are the Stochastic Gradient Descent (SGD), with vanilla (vSGD), with momentum (SGDm), with momentum and nesterov (SGDm+n)), Root Mean Square Propagation (RMSProp), Adaptive Moment Estimation (Adam), Adaptive Gradient (AdaGrad), Adaptive Delta (AdaDelta), Adaptive moment estimation Extension based on infinity norm (Adamax) and Nesterov-accelerated Adaptive Moment Estimation (Nadam). We trained the model and evaluated the optimization techniques in terms of convergence speed, accuracy and loss function using three randomly selected publicly available image classification datasets. The overall experimental results obtained show Nadam achieved better performance across the three datasets in comparison to the other optimization techniques, while AdaDelta performed the worst.

KW - Artificial Intelligence

KW - deep learning

KW - optimizers

KW - performance measures

KW - stochastic gradient descent

UR - http://www.scopus.com/inward/record.url?scp=85070414431&partnerID=8YFLogxK

U2 - 10.1109/CTEMS.2018.8769211

DO - 10.1109/CTEMS.2018.8769211

M3 - Conference contribution

AN - SCOPUS:85070414431

T3 - Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018

SP - 92

EP - 99

BT - Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018

A2 - Niranjan, S. K.

A2 - Desai, Veena

A2 - Rajpurohit, Vijay S.

A2 - Nadkatti, M N

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 1st International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018

Y2 - 21 December 2018 through 23 December 2018

ER -

Dogo EM, Afolabi OJ, Nwulu NI, Twala B, Aigbavboa CO. A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks. In Niranjan SK, Desai V, Rajpurohit VS, Nadkatti MN, editors, Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 92-99. 8769211. (Proceedings of the International Conference on Computational Techniques, Electronics and Mechanical Systems, CTEMS 2018). doi: 10.1109/CTEMS.2018.8769211

A Comparative Analysis of Gradient Descent-Based Optimization Algorithms on Convolutional Neural Networks

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this