Parallel Aspects of OpenFOAM with Large Eddy Simulations

Open FOAM is a mainstream open-source frame-work for the simulation in several areas of CFD and engineering whose syntax is a high level representation of the mathematical notation of physical models, internal details like parallelization, tensor algebra, and mesh manipulation are hidden and automatically integrated. We used the back-facing step geometry with Large Eddy Simulations and semi-implicit methods to investigate the scalability and important MPI characteristics of Open FOAM. Moreover, this geometry provides a configuration with representative features found in current engineering and HPC problems. The algebraic multigrid solver, for example, was found to be a very powerful and fast iterative solver for the solution of PDEs compared to the BiGC during the weak and strong scaling tests. It was also determined strong relations between the sizes and shapes of sud domains with the footprint of the inter-domains communications subroutines using a graph-based partitioner. Thus, setting the bases for optimal domain decompositions. However, it was also found that the master-slave strategy introduces an unexpected bottleneck in the communication of scalar values when more than a hundred of MPI tasks were deployed. An extensive analysis revealed that this anomaly was present in few MPI tasks resulting in an severe performance reduction. Finally, we highlight the importance of tracing and profiling tools, IPM is a novel implementation that could instrument Open FOAM successfully.