Title :
Single stream parallelization of generalized LSTM-like RNNs on a GPU
Author :
Kyuyeon Hwang ; Wonyong Sung
Author_Institution :
Dept. of Electr. & Comput. Eng., Seoul Nat. Univ., Seoul, South Korea
Abstract :
Recurrent neural networks (RNNs) have shown outstanding performance on processing sequence data. However, they suffer from long training time, which demands parallel implementations of the training procedure. Parallelization of the training algorithms for RNNs are very challenging because internal recurrent paths form dependencies between two different time frames. In this paper, we first propose a generalized graph-based RNN structure that covers the most popular long short-term memory (LSTM) network. Then, we present a parallelization approach that automatically explores parallelisms of arbitrary RNNs by analyzing the graph structure. The experimental results show that the proposed approach shows great speed-up even with a single training stream, and further accelerates the training when combined with multiple parallel training streams.
Keywords :
data handling; graph theory; graphics processing units; learning (artificial intelligence); recurrent neural nets; GPU; generalized LSTM-like RNN; generalized graph-based RNN structure; graphics processing unit; long short-term memory network; multiple parallel training streams; recurrent neural networks; sequence data processing; single stream parallelization; single training stream; Additives; Graphics processing units; Logic gates; Parallel processing; Recurrent neural networks; Training; Recurrent neural network (RNN); generalization; graphics processing unit (GPU); long short-term memory (LSTM); parallelization;
Conference_Titel :
Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on
DOI :
10.1109/ICASSP.2015.7178129
