A Novel Parallel Architecture with Fault-Tolerance for Joining Bi-directional Data Streams in Cloud

Nowadays, there are more and more data sources, such as online stores, wireless sensors, and stock tickers, etc., which produce a mass of data as unbounded streams. Investigating the potential information from these rich statistical data in real time generates enormous values. In most cases, it is essential to join two or more data streams in order to find sufficient information. However, as the volume of these data becomes larger, joining them in a single machine is not an effective way. One of the most practical solutions is to use a shared-nothing cluster built on a cloud to deal with these data streams. In this paper we provide a novel parallel architecture named Dual-Assembly-Pipeline(DAP) with fault-tolerance, in which we join bi-directional data streams by considering the processing nodes´ failures. Especially, virtual machines in a cloud may fail so that fault-tolerance becomes more and more important when users issue a continuous query. Our experiments show that our DAP model can effectively join bi-directional data streams in parallel. What´s more, when non-adjacent virtual machines fail at the same time, all data can be recovered, while adjacent ones fail, only parts of data can be reinstated.