A Large-scale and Decentralized Infrastructure for Multiple Queries Optimization and Aggregation

Leveraging DHTs (distributed hash table), we propose a novel architecture, which applies multiple query optimization technique to efficiently aggregate queries in large scale distributed networks. We target applications that continuously query the data sources and manipulate a large amount of query results. Our goal is to implement system components to aggregate distributed queries in the network, so as to (1) reduce the overhead (CPU cycles, disk I/O etc.) on the data source nodes for query evaluation; (2) save the overall network bandwidth cost for delivering queries and the query results. To deal with the skewed load distribution, we also provide load balancing mechanisms to ensure that no node in the system is unduly loaded. The simulation results show that the proposed architecture can efficiently distribute the query processing in the network and significantly reduce the number of queries evaluated on the data source nodes. The network bandwidth consumption is largely reduced by eliminating the transmission of common data items.