Towards a Heterogeneous Compute Engine for Software Acceleration on Big Data Infrastructures

Big Data processing requires significant amount of computing resources that are typically distributed in traditional clusters or modern cloud infrastructures. Software stacks, such as Hadoop, have been evolved for managing distributed resources, schedule workloads, support data storage and sharing. However, these stacks focus on homogeneous node architectures and only consider CPU processors while ignoring the popular trend of computational accelerators, such as GPUs. The computational power and energy efficiency of accelerators make them desirable components for Big Data infrastructures. This paper addresses this issue and enables accelerator integration in Big Data software stacks. We present preliminary work on the design and implementation of a Heterogeneous Compute Engine that allows heterogeneous parallel processing on Big Data infrastructures. Our work seamlessly integrates with Hadoop and enables computation acceleration.