Where should the threads go? Leveraging hierarchical data locality to solve the thread affinity dilemma

We are proposing a novel framework that ameliorates locality-aware parallel programming models, by defining a hierarchical data locality model extension. We also propose two hierarchical thread partitioning algorithms. These algorithms synthesize hierarchical thread placement layouts that targets minimizing the program´s overall communication costs. We demonstrate the effectiveness of our approach using the NAS Parallel Benchmarks implemented in Unified Parallel C (UPC) using a modified Berkeley UPC Compiler and runtime system. We achieved performance gains of up to 88% in performance by applying the placement layouts our algorithms suggest.