Accelerating Neural Network Training with FSGQR: A Scalable and High-Performance Alternative to Adam

[1] Oveis Abedinia, Nima Amjady, and Noradin Ghadimi. Solar energy forecasting based on hybrid neural network and improved meta-heuristic algorithm. Computational Intelligence, 34(1):241–260, 2018. Search in Google Scholar

[2] U. Rajendra Acharya, Shu Lih Oh, Yuki Hagiwara, Jen Hong Tan, and Hojjat Adeli. Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals. Computers in Biology and Medicine, 100:270–278, 2018. Search in Google Scholar

[3] Igor Aizenberg, Dmitriy V. Paliy, Jacek M. Zurada, and Jaakko T. Astola. Blur identification by multilayer neural network based on multi-valued neurons. IEEE Transactions on Neural Networks, 19(5):883–898, 2008. Search in Google Scholar

[4] E. Angelini, G. di Tollo, and A. Roli. A neural network approach for credit risk evaluation. The Quarterly Review of Economics and Finance, 48(4):733–755, 2008. Search in Google Scholar

[5] Jarosław Bilski. Struktury równoległe dla jednokierunkowych i dynamicznych sieci neuronowych. Akademicka Oficyna Wydawnicza EXIT, 2013. Search in Google Scholar

[6] Jarosław Bilski and Alexander I. Galushkin. A new proposition of the activation function for significant improvement of neural networks performance. In Artificial Intelligence and Soft Computing, volume 9602 of Lecture Notes in Computer Science, pages 35–45. Springer-Verlag Berlin Heidelberg, 2016. Search in Google Scholar

[7] Jarosław Bilski and Leszek Rutkowski. A fast training algorithm for neural networks. IEEE Transaction on Circuits and Systems Part II, 45(6):749–753, 1998. Search in Google Scholar

[8] Jarosław Bilski and Jacek Smoląg. Parallel realisation of the recurrent multi layer perceptron learning. Artificial Intelligence and Soft Computing, Springer-Verlag Berlin Heidelberg, (LNAI 7267):12–20, 2012. Search in Google Scholar

[9] Jarosław Bilski and Jacek Smoląg. Parallel approach to learning of the recurrent Jordan neural network. Artificial Intelligence and Soft Computing, Springer-Verlag Berlin Heidelberg, (LNAI 7895):32–40, 2013. Search in Google Scholar

[10] Jarosław Bilski and Jacek Smoląg. Parallel architectures for learning the RTRN and Elman dynamic neural network. IEEE Transactions on Parallel and Distributed Systems, 26(9):2561–2570, 2015. Search in Google Scholar

[11] Jarosław Bilski, Jacek Smoląg, and Alexander I. Galushkin. The parallel approach to the conjugate gradient learning algorithm for the feed-forward neural networks. In Artificial Intelligence and Soft Computing, volume 8467 of Lecture Notes in Computer Science, pages 12–21. Springer-Verlag Berlin Heidelberg, 2014. Search in Google Scholar

[12] Jarosław Bilski, Jacek Smoląg, and Jacek M. Zurada. Parallel approach to the Levenberg-Marquardt learning algorithm for feedforward neural networks. In Artificial Intelligence and Soft Computing, volume 9119 of Lecture Notes in Computer Science, pages 3–14. Springer-Verlag Berlin Heidelberg, 2015. Search in Google Scholar

[13] Jarosław Bilski, Bartosz Kowalczyk, and Andrzej Cader. Modifications of the Givens training algorithm for artificial neural networks. In Leszek Rutkowski, Rafał Scherer, Marcin Korytkowski, Witold Pedrycz, Ryszard Tadeusiewicz, and Jacek M. Zurada, editors, Artificial Intelligence and Soft Computing, pages 14–28, Cham, 2019. Springer International Publishing. Search in Google Scholar

[14] Jarosław Bilski, Bartosz Kowalczyk, Marek Kisiel-Dorohinicki, Agnieszka Siwocha, and Jacek Żurada. Towards a very fast feedforward multilayer neural networks training algorithm. Journal of Artificial Intelligence and Soft Computing Research, 12(3):181–195, 2022. Search in Google Scholar

[15] Jarosław Bilski, Bartosz Kowalczyk, Andrzej Marjański, Michał Gandor, and Jacek Zurada. A novel fast feedforward neural networks training algorithm. Journal of Artificial Intelligence and Soft Computing Research, 11(4):287–306, 2021. Search in Google Scholar

[16] Jarosław Bilski, Jacek Smoląg, Bartosz Kowalczyk, Konrad Grzanek, and Ivan Izonin. Fast computational approach to the Levenberg-Marquardt algorithm for training feedforward neural networks. Journal of Artificial Intelligence and Soft Computing Research, 12(2):45–61, 2023. Search in Google Scholar

[17] W. Duch, K. Swaminathan, and J. Meller. Artificial intelligence approaches for rational drug design and discovery. Current Pharmaceutical Design, 13(14):1497–1508, 2007. Search in Google Scholar

[18] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159, 07 2011. Search in Google Scholar

[19] Marcin Gabryel, Eliza Kocić, Milan Kocić, Zofia Patora-Wysocka, Min Xiao, and Mirosław Pawlak. Accelerating user profiling in e-commerce using conditional GAN networks for synthetic data generation. Journal of Artificial Intelligence and Soft Computing Research, 14(4):309–319, 2024. Search in Google Scholar

[20] Morven W. Gentleman. Least Squares Computations by Givens Transformations Without Square Roots. IMA Journal of Applied Mathematics, 12(3):329–336, 12 1973. Search in Google Scholar

[21] Ghosh and Reilly. Credit card fraud detection with a neural-network. In 1994 Proceedings of the Twenty-Seventh Hawaii International Conference on System Sciences, volume 3, pages 621–630, Jan 1994. Search in Google Scholar

[22] Wallace Givens. Computation of plain unitary rotations transforming a general matrix to triangular form. Journal of The Society for Industrial and Applied Mathematics, 6:26–50, 1958. Search in Google Scholar

[23] J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, and T. Chen. Recent advances in convolutional neural networks. Pattern Recognition, 77:354–377, 2018. Search in Google Scholar

[24] Martin T. Hagan and Mohammad B. Menhaj. Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neuralnetworks, 5:989–993, 1994. Search in Google Scholar

[25] A. Horzyk and R. Tadeusiewicz. Self-optimizing neural networks. In Fu-Liang Yin, Jun Wang, and Chengan Guo, editors, Advances in Neural Networks – ISNN 2004, pages 150–155, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg. Search in Google Scholar

[26] Andrzej Kiełbasiński and Hubert Schwetlick. Numeryczna Algebra Liniowa: Wprowadzenie do Obliczeń Zautomatyzowanych. Wydawnictwa Naukowo-Techniczne, Warszawa, 1992. Search in Google Scholar

[27] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. Search in Google Scholar

[28] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. Search in Google Scholar

[29] Dominik Lewy and Jacek Mańdziuk. Training CNN classifiers solely on webly data. Journal of Artificial Intelligence and Soft Computing Research, 13(1):75–92, 2023. Search in Google Scholar

[30] Y. Li, R. Cui, Z. Li, and D. Xu. Neural network approximation based near-optimal motion planning with kinodynamic constraints using RRT. IEEE Transactions on Industrial Electronics, 65(11):8718–8729, Nov 2018. Search in Google Scholar

[31] H. Liu, X. Mi, and Y. Li. Wind speed forecasting method based on deep learning strategy using empirical wavelet transform, long short term memory neural network and Elman neural network. Energy Conversion and Management, 156:498–514, 2018. Search in Google Scholar

[32] M. Mazurowski, P. Habas, J. Zurada, J. Lo, J. Baker, and G. Tourassi. Training neural network classifiers for medical decision making: The effects of imbalanced datasets on classification performance. Neural networks : the official journal of the International Neural Network Society, 21:427–36, 03 2008. Search in Google Scholar

[33] Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1995. DOI: https://doi.org/10.24432/C55C7W. Search in Google Scholar

[34] Tacjana Niksa-Rynkiewicz, Piotr Stomma, Anna Witkowska, Danuta Rutkowska, Adam Słowik, Krzysztof Cpałka, Joanna Jaworek-Korjakowska, and Piotr Kolendo. An intelligent approach to short-term wind power prediction using deep neural networks. Journal of Artificial Intelligence and Soft Computing Research, 13(3):197–210, 2023. Search in Google Scholar

[35] B.T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964. Search in Google Scholar

[36] Nataliya Shakhovska, Andrii Shebeko, and Yarema Prykarpatskyy. A novel explainable AI model for medical data analysis. Journal of Artificial Intelligence and Soft Computing Research, 14(2):121–137, 2024. Search in Google Scholar

[37] R. Shirin. A neural network approach for retailer risk assessment in the aftermarket industry. Benchmarking: An International Journal, 26(5):1631–1647, Jan 2019. Search in Google Scholar

[38] A.K. Singh, S.K. Jha, and A.V. Muley. Candidates selection using artificial neural network technique in a pharmaceutical industry. In Siddhartha Bhattacharyya, Aboul Ella Hassanien, Deepak Gupta, Ashish Khanna, and Indrajit Pan, editors, International Conference on Innovative Computing and Communications, pages 359–366, Singapore, 2019. Springer Singapore. Search in Google Scholar

[39] R. Tadeusiewicz, L. Ogiela, and M.R. Ogiela. Cognitive analysis techniques in business planning and decision support systems. In L. Rutkowski, R. Tadeusiewicz, L.A. Zadeh, and J.M. Zurada, editors, Artificial Intelligence and Soft Computing – ICAISC 2006, pages 1027–1039, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. Search in Google Scholar

[40] K.Y. Tam and M. Kiang. Predicting bank failures: A neural network approach. Applied Artificial Intelligence, 4(4):265–282, 1990. Search in Google Scholar

[41] Paul Werbos. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Harvard University, 1974. Search in Google Scholar

[42] B.M. Wilamowski. Neural network architectures and learning algorithms. IEEE Industrial Electronics Magazine, 3(4):56–63, 2009. Search in Google Scholar

[43] I-Cheng Yeh. Concrete Compressive Strength. UCI Machine Learning Repository, 2007. DOI: https://doi.org/10.24432/C5PK67. Search in Google Scholar

[44] Matthew D. Zeiler. Adadelta: An adaptive learning rate method, 2012. Search in Google Scholar

[45] Junming Zhang, Hao Dong, Jinfeng Gao, Ruxian Yao, Gangqiang Li, and Haitao Wu. Self-organized operational neural networks for the detection of atrial fibrillation. Journal of Artificial Intelligence and Soft Computing Research, 14(1):63–75, 2024. Search in Google Scholar