Title :
SciSpark: Applying in-memory distributed computing to weather event detection and tracking
Author :
Rahul Palamuttam;Renato Marroqu?n Mogrovejo;Chris Mattmann;Brian Wilson;Kim Whitehall;Rishi Verma;Lewis McGibbney;Paul Ramirez
Author_Institution :
NASA Jet Propulsion Laboratory California Institute of Technology Pasadena, CA, USA
Abstract :
In this paper we present SciSpark, a Big Data framework that extends Apache™ Spark for scaling scientific computations. The paper details the initial architecture and design of SciSpark. We demonstrate how SciSpark achieves parallel ingesting and partitioning of earth science satellite and model datasets. We also illustrate the usability and extensibility of SciSpark by implementing aspects of the Grab ´em Tag ´em Graph ´em (GTG) algorithm using SciSpark and its Map Reduce capabilities. GTG is a topical automated method for identifying and tracking Mesoscale Convective Complexes in satellite infrared datasets.
Keywords :
"Arrays","Sparks","Libraries","Clouds","Meteorology","File systems"
Conference_Titel :
Big Data (Big Data), 2015 IEEE International Conference on
DOI :
10.1109/BigData.2015.7363983