Scalable Clustering Using PACT Programming Model

Recent spurt in research related to scalability of data mining algorithms can be attributed to advances in cloud computing technology, which enables data-intensive applications in distributed environment. Map-Reduce has been the most popular programming paradigm for developing applications in large scale distributed environments. In this paper we present design of a scalable clustering algorithm ´Exclusive and Complete Clustering using PACT Programming model´ (ExCC-P) for recently developed Stratosphere system for cloud computing environment. This system supports novel model for programming in large scale distributed environments. Based on the concept of Parallelization Contracts, the PACT programming model is a generalization of Map-Reduce paradigm. PACT programs are complied by a PACT compiler and executed by Nephele execution engine of Stratosphere after optimizing data-flow graphs. The algorithm ExCC-P is proposed as a solution for incremental clustering of unbounded massive data sets, to be executed in Stratosphere environment. The algorithm discretizes data space into a conceptual grid and processes data in batches. After a batch is processed, the algorithm applies connected component analysis on the grid to deliver arbitrarily shaped clusters. Limited experimentation on this under-development system (Stratosphere) validated PACT programming model for the proposed algorithm.