<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>In this paper<sup>1</sup> a new neural networks training algorithm is presented. The algorithm originates from the Recursive Least Squares (RLS) method commonly used in adaptive filtering. It uses the QR decomposition in conjunction with the Givens rotations for solving a normal equation - resulting from minimization of the loss function. An important parameter in neural networks is training time. Many commonly used algorithms require a big number of iterations in order to achieve a satisfactory outcome while other algorithms are effective only for small neural networks. The proposed solution is characterized by a very short convergence time compared to the well-known backpropagation method and its variants. The paper contains a complete mathematical derivation of the proposed algorithm. There are presented extensive simulation results using various benchmarks including function approximation, classification, encoder, and parity problems. Obtained results show the advantages of the featured algorithm which outperforms commonly used recent state-of-the-art neural networks training algorithms, including the Adam optimizer and the Nesterov’s accelerated gradient.</p>
</abstract>

In this paper1 a new neural networks training algorithm is presented. The algorithm originates from the Recursive Least Squares (RLS) method commonly used in adaptive filtering. It uses the QR decomposition in conjunction with the Givens rotations for solving a normal equation - resulting from minimization of the loss function. An important parameter in neural networks is training time. Many commonly used algorithms require a big number of iterations in order to achieve a satisfactory outcome while other algorithms are effective only for small neural networks. The proposed solution is characterized by a very short convergence time compared to the well-known backpropagation method and its variants. The paper contains a complete mathematical derivation of the proposed algorithm. There are presented extensive simulation results using various benchmarks including function approximation, classification, encoder, and parity problems. Obtained results show the advantages of the featured algorithm which outperforms commonly used recent state-of-the-art neural networks training algorithms, including the Adam optimizer and the Nesterov’s accelerated gradient.

A Novel Fast Feedforward Neural Networks Training Algorithm

Department of Intelligent Computer Systems

Faculty of Computer Science and Telecommunications

Department of Computer and Electrical Engineering

Journal of Artificial Intelligence and Soft Computing Research

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

{"article-title":"A Novel Fast Feedforward Neural Networks Training Algorithm"}

In this paper1 a new neural networks training algorithm is presented. The algorithm originates from the Recursive Least Squares (RLS) method commonly used in...

A Novel Fast Feedforward Neural Networks Training Algorithm

Publié en ligne: 08 oct. 2021

Pages: 287 - 306

Reçu: 15 févr. 2021

Accepté: 24 juil. 2021

DOI: https://doi.org/10.2478/jaiscr-2021-0017

Mots clés
neural network training algorithm, QR decomposition, Givens rotations, approximation, classification

© 2021 Jarosław Bilski et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

A Novel Fast Feedforward Neural Networks Training Algorithm

Publié en ligne: 08 oct. 2021

Pages: 287 - 306

Reçu: 15 févr. 2021

Accepté: 24 juil. 2021

DOI: https://doi.org/10.2478/jaiscr-2021-0017

Mots clésneural network training algorithm, QR decomposition, Givens rotations, approximation, classification

© 2021 Jarosław Bilski et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Mots clés
neural network training algorithm, QR decomposition, Givens rotations, approximation, classification