DocumentCode :
3717155
Title :
A predictive scheduling framework for fast and distributed stream data processing
Author :
Teng Li;Jian Tang;Jielong Xu
Author_Institution :
Department of Electrical Engineering and Computer Science at Syracuse University
fYear :
2015
Firstpage :
333
Lastpage :
338
Abstract :
In a distributed stream data processing system, an application is usually modeled using a directed graph, in which each vertex corresponds to a data source or a processing unit, and edges indicate data flow. In this paper, we propose a novel predictive scheduling framework to enable fast and distributed stream data processing, which features topology-aware performance prediction and predictive scheduling. For prediction, we present a topology-aware method to accurately predict the average tuple processing time of an application for a given scheduling solution, according to the topology of the application graph and runtime statistics. For scheduling, we present an effective algorithm to assign threads to machines under the guidance of prediction results. To validate and evaluate the proposed framework, we implemented it based on a highly-regarded distributed stream data processing platform, Storm, and tested it with two representative applications: word count (stream version) and log stream processing. Extensive experimental results show (1) The topology-aware prediction method offers an average accuracy of 83.7%. (2) The predictive scheduling framework reduces the average tuple processing time by 25.9% on average, compared to Storm´s default scheduler.
Keywords :
"Instruction sets","Prediction algorithms","Scheduling","Data processing","Topology","Predictive models","Distributed databases"
Publisher :
ieee
Conference_Titel :
Big Data (Big Data), 2015 IEEE International Conference on
Type :
conf
DOI :
10.1109/BigData.2015.7363773
Filename :
7363773
Link To Document :
بازگشت