An intelligent ETL workflow framework based on data partition

ETL tool is an important part to build a data warehouse and data centers. For massive data processing, this paper presents an intelligent ETL workflow framework based on the distributed computing servers, adding an intelligent manipulative module, acquiring the data of the system efficiency and resources, operating data, dynamically adjusting the ETL strategy, and doing corresponding data segmentation for larger jobs, realizing workflow optimization for multi-machine parallel execution, improving operational efficiency, and facilitating error recovery. Intelligent control module is composed of the monitor, knowledge base, and the selector. The source data horizontal partition is the basis and difficulty to achieve multi-machine parallel.