An Inter-framework Cache for Diverse Data-Intensive Computing Environments

Hadoop Distributed File System (HDFS) provides the storage to keep analyzing outcomes for the diversity of frameworks. MapReduce, Storm, and Spark each applies on batching, streaming and in-memory computing, all of them need the HDFS to collect and assemble results. For coping with Big-Data analysis in the real world, complicated platforms required working together. However, collaborating analysis on heterogeneous frameworks, the data must be write-through firstly and post-fetch upon HDFS that degrades the performance and lower the effectiveness of the whole system. For best our knowledge, no previous work had focused on inter-framework data caching. To solve above problems on collaborating analysis within heterogeneous frameworks such as Hadoop and Strom, in this paper, we propose a cache system upon YARN called "Inter-Framework Cache" (IF-cache). It uses in-memory cache to reserve temporary outcomes while also reducing the HDFS access frequency and improve analysis performance. Experiments had shown that Hadoop with IF-cache can reduce about 50% times comparing the no-cache one.