A Distributed-Memory Parallelization of a Shared-Memory Parallel Ensemble Kalman Filter

Inverse problems arise in various areas of science and engineering. These problems are not only difficult to solve numerically, but they also require a large amount of computer resources both in time and memory. It is therefore not surprising that inverse problems are often solved using techniques from high-performance computing. We consider the parallelization of an inverse problem in the field of geothermal reservoir engineering. In this particular scientific application, the underlying software package is already parallelized using the shared-memory programming paradigm Open MP. Here, we present an extension of this parallelization to distributed memory enabling a hybrid Open MP/MPI parallelization. The situation is different from the standard way of hybrid parallel programming because the data structures of the Open MP-parallelized code differ from those in the serial implementation. We exploit this transformation of the data structures in our distributed-memory strategy for parallelizing an ensemble Kalman filter, a particular method for the solution of inverse problems. We describe this novel parallelization strategy, introduce a performance model, and present timing results on a compute cluster using nodes with 2 sockets, each equipped with Intel Xeon X5675 Westmere EP processors with 6 cores. All timing results are obtained with a pure MPI parallelization without using any Open MP threads.