An MAS-based and fault-tolerant distributed ETL workflow engine

To improve the stability and reliability of the ETL workflow engine, in this paper, we put forward a new MAS-based and fault-tolerant distributed ELT workflow engine which is used to study the log management and exception handling mechanism in the distributed computing environment. In the new engine, the ETL jobs will be split into several job domains firstly. Secondly, when an error happens in a job domain, the engine will catch the exception from the error logs and then invoke our new recovery algorithm, Batch, to recover the job domain. The experiment result shows that our engine is not only much more stabile and reliable but also more efficiency in the execution of ETL workflow.