DM-Midware: A Middleware to Enable High Performance Data Mining in Heterogeneous Cloud

Cloud computing has become a popular high performance computing model where resources are provided as services over the Web. Users are starting to adopt cloud model in data mining applications. However, due to the complexity of parallel/cloud computing, it is difficult for average users to express a parallel computing paradigm for their applications in cloud. In order to isolate users from the complexity of parallel/cloud programming, a middleware to enable high performance data mining, called DM-Midware, is proposed. It hides the details of MapReduce programming from users by automatically launching mappers through a set of user programming APIs. Directive-based parallelization scheme automatically "translates" a serial program into a SMP or Multi-core based parallel program. Heterogeneous computing resources can be invoked to conduct parallel execution by API-based scheme. A two-step scheduling scheme is proposed to maximize the throughput of the cloud system. We evaluate DM-Midware by executing a representative data mining algorithm in a private cloud. Experimental results demonstrate good scalability and adaptability.