Title :
Research of distributed ETL engine based on MAS and data partition
Author :
Wang, Guiteng ; Guo, Chaozhen
Author_Institution :
Coll. of Math. & Comput. Sci., Fuzhou Univ., Fu Zhou, China
Abstract :
In order to improve the efficiency of ETL workflow executing, this paper presents a distributed ETL engine based on MAS and data partition technology and also researches the methods of partitioning the massive data stream in both horizontal and vertical ways. The engine referred to will partition an ETL workflow which meets the conditions of being partitioned into multiple sub workflows for parallel executing. Each of the sub workflow is executed by an agent, so that multiple agents could work together to complete the collaborative work. Experimental results show that this system has good scalability and could well improve the efficiency of ETL workflow executing.
Keywords :
enterprise resource planning; multi-agent systems; parallel processing; workflow management software; MAS; data partition; distributed ETL engine; multiple sub workflows; parallel execution; Computer architecture; Corporate acquisitions; Distributed databases; Engines; Registers; Servers; Sockets; CSCW; ETL workflow; MAS; data partition; distribute ETL;
Conference_Titel :
Computer Supported Cooperative Work in Design (CSCWD), 2011 15th International Conference on
Conference_Location :
Lausanne
Print_ISBN :
978-1-4577-0386-7
DOI :
10.1109/CSCWD.2011.5960096
