On the Performance of Parallel Neural Network Implementations on Distributed Memory Architectures

In this paper, we study the impact of multi processor memory systems in particular, the distributed memory (DM) and virtual shared memory (VSM), on the implementation of parallel backpropagation neural network algorithms. In the first instance, neural network is partitioned into sub neural networks by applying a hybrid partitioning scheme. In the second, each partitioned network is evaluated with matrix multiplication. Three different sizes of neural networks are used and exchange rate prediction used as a reference problem. Parallel implementations for each of the distributed memory and virtual shared memory scenarios is obtained. These algorithms are implemented on a high performance cluster, "Monolith" consisting of over 396 nodes. Programming is realized using Message Passing Interface (MPI) library and C-Linda. The partitioned, matrix multiplication has the fastest execution time, and DM/MPI implementation is always faster than the VSM/Linda equivalent. However in VSM/Linda it is possible to allow the parallel neural network to choose the optimum number of processors dynamically.