Title :
GoldRush: Resource efficient in situ scientific data analytics using fine-grained interference aware execution
Author :
Fang Zheng ; Hongfeng Yu ; Hantas, Can ; Wolf, Michael ; Eisenhauer, Greg ; Schwan, Karsten ; Abbasi, Hasan ; Klasky, Scott
Author_Institution :
Georgia Inst. of Technol., Atlanta, GA, USA
Abstract :
Severe I/O bottlenecks on High End Computing platforms call for running data analytics in situ. Demonstrating that there exist considerable resources in compute nodes un-used by typical high end scientific simulations, we leverage this fact by creating an agile runtime, termed GoldRush, that can harvest those otherwise wasted, idle resources to efficiently run in situ data analytics. GoldRush uses fine-grained scheduling to “steal” idle resources, in ways that minimize interference between the simulation and in situ analytics. This involves recognizing the potential causes of on-node resource contention and then using scheduling methods that prevent them. Experiments with representative science applications at large scales show that resources harvested on compute nodes can be leveraged to perform useful analytics, significantly improving resource efficiency, reducing data movement costs incurred by alternate solutions, and posing negligible impact on scientific simulations.
Keywords :
Big Data; data analysis; scheduling; Big Data; GoldRush system; I/O bottlenecks; data movement cost reduction; fine-grained interference aware execution; fine-grained scheduling method; high end computing platforms; high end scientific simulations; idle resources; on-node resource contention; resource efficient in situ scientific data analytics; Abstracts; Context; Instruction sets; Lead; Programmable logic arrays; Runtime; Switches; Design; Measurement; Performance;
Conference_Titel :
High Performance Computing, Networking, Storage and Analysis (SC), 2013 International Conference for
Conference_Location :
Denver, CO
Print_ISBN :
978-1-4503-2378-9
DOI :
10.1145/2503210.2503279