Employing in-memory data grids for distributed graph processing

In-memory data grid (IMDG) is a new technology that enables scalable and low-latency processing of big data by sharding it over the RAMs of multiple servers. In this paper, we explore the design space of IMDGs to identify their advantages and avoid their drawbacks. We present the performance tradeoffs of IMDGs using unit tests on core distributed operations and data structures. For evaluation, we use large-scale graph processing, a challenging task that requires a high degree of communication and coordination between vertices. We find that while IMDGs cannot compete with specialized distributed frameworks (such as Giraph and GraphLab) for batch-mode graph processing, they excel for online graph processing and offer exciting opportunities for social networks and web services applications.