Title :
Partitioning real-time ETL workflows
Author :
Simitsis, Alkis ; Gupta, Chetan ; Wang, Song ; Dayal, Umeshwar
Author_Institution :
HP Labs., Palo Alto, CA, USA
Abstract :
Many organizations are aiming to move away from traditional batch processing ETL to real-time ETL (RT-ETL). This move is motivated by a need to analyze and take decisions on as fresh a data as possible. The RT-ETL engines operate on the abstraction of data flow executed on parallel architectures. For high throughput and low response times, there is a need for partitioning the data over the large number of nodes in the engine. In this paper, we consider the problem of partitioning realtime ETL flows and we propose a high level architecture for that.
Keywords :
batch processing (computers); data flow computing; workflow management software; batch processing; data flow execution; high level architecture; parallel architectures; real-time ETL workflows; Costs; Data warehouses; Decision making; Delay; Design optimization; Fault tolerance; Humans; Maintenance; Merging; Real time systems;
Conference_Titel :
Data Engineering Workshops (ICDEW), 2010 IEEE 26th International Conference on
Conference_Location :
Long Beach, CA
Print_ISBN :
978-1-4244-6522-4
Electronic_ISBN :
978-1-4244-6521-7
DOI :
10.1109/ICDEW.2010.5452754
