<abstract xmlns="http://www.w3.org/1999/xhtml"><p>Recurrent neural networks (RNN) have been successfully applied to various sequential decision-making tasks, natural language processing applications, and time-series predictions. Such networks are usually trained through back-propagation through time (BPTT) which is prohibitively expensive, especially when the length of the time dependencies and the number of hidden neurons increase. To reduce the training time, extreme learning machines (ELMs) have been recently applied to RNN training, reaching a 99% speedup on some applications. Due to its non-iterative nature, ELM training, when parallelized, has the potential to reach higher speedups than BPTT.</p><p>In this work, we present Opt-PR-ELM, an optimized parallel RNN training algorithm based on ELM that takes advantage of the GPU shared memory and of parallel QR factorization algorithms to efficiently reach optimal solutions. The theoretical analysis of the proposed algorithm is presented on six RNN architectures, including LSTM and GRU, and its performance is empirically tested on ten time-series prediction applications. Opt-PR-ELM is shown to reach up to 461 times speedup over its sequential counterpart and to require up to 20x less time to train than parallel BPTT. Such high speedups over new generation CPUs are extremely crucial in real-time applications and IoT environments.</p></abstract>

Recurrent neural networks (RNN) have been successfully applied to various sequential decision-making tasks, natural language processing applications, and time-series predictions. Such networks are usually trained through back-propagation through time (BPTT) which is prohibitively expensive, especially when the length of the time dependencies and the number of hidden neurons increase. To reduce the training time, extreme learning machines (ELMs) have been recently applied to RNN training, reaching a 99% speedup on some applications. Due to its non-iterative nature, ELM training, when parallelized, has the potential to reach higher speedups than BPTT.In this work, we present Opt-PR-ELM, an optimized parallel RNN training algorithm based on ELM that takes advantage of the GPU shared memory and of parallel QR factorization algorithms to efficiently reach optimal solutions. The theoretical analysis of the proposed algorithm is presented on six RNN architectures, including LSTM and GRU, and its performance is empirically tested on ten time-series prediction applications. Opt-PR-ELM is shown to reach up to 461 times speedup over its sequential counterpart and to require up to 20x less time to train than parallel BPTT. Such high speedups over new generation CPUs are extremely crucial in real-time applications and IoT environments.

An Optimized Parallel Implementation of Non-Iteratively Trained Recurrent Neural Networks

Department of Electrical and Computer Engineering

Journal of Artificial Intelligence and Soft Computing Research

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Recurrent neural networks (RNN) have been successfully applied to various sequential decision-making tasks, natural language processing applications, and...

An Optimized Parallel Implementation of Non-Iteratively Trained Recurrent Neural Networks

Published Online: Dec 03, 2020

Page range: 33 - 50

Received: May 07, 2020

Accepted: Sep 14, 2020

DOI: https://doi.org/10.2478/jaiscr-2021-0003

Keywords
GPU implementation, parallelization, Recurrent Neural Network (RNN), Long-short Term Memory (LSTM), Gated Recurrent Unit (GRU), Extreme Learning Machines (ELM), non-iterative training

© 2021 Julia El Zini et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

An Optimized Parallel Implementation of Non-Iteratively Trained Recurrent Neural Networks

Published Online: Dec 03, 2020

Page range: 33 - 50

Received: May 07, 2020

Accepted: Sep 14, 2020

DOI: https://doi.org/10.2478/jaiscr-2021-0003

KeywordsGPU implementation, parallelization, Recurrent Neural Network (RNN), Long-short Term Memory (LSTM), Gated Recurrent Unit (GRU), Extreme Learning Machines (ELM), non-iterative training

© 2021 Julia El Zini et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Keywords
GPU implementation, parallelization, Recurrent Neural Network (RNN), Long-short Term Memory (LSTM), Gated Recurrent Unit (GRU), Extreme Learning Machines (ELM), non-iterative training