A Parallel Data Processing Middleware Based on Clusters

HPDPM is a middleware system applying in share nothing clusters architecture to support parallel and distributed computing. Presenting a new method to use parallel data processing middleware instead of parallel database system provides the ability for high performance computing. A framework is given for realizing parallel data manipulation. The primary modules of the middleware are described. Key techniques used to improve system performance, include data placement and semantic caching are discussed in detail. Then, the work principles and work steps of the middleware are presented. Implementation and experiments of this study showed that this approach can improve system performance efficiently. At present, the middleware system has been applied to some large engineering projects which capacity of data is a little more than 1000 Gigabytes.