An Architecture for Orchestrating Hadoop Applications in Hybrid Cloud

MapReduce is a programming model for processing and generating large data sets, and Hadoop, a MapReduce implementation, is a good tool to handle Big Data. Cloud computing with its ubiquitous characteristic, on demand and dynamic resource provisioning at low cost has potential to be the environment to treat big data. However, using Hadoop on the cloud spends time and requires technical knowledge from users. The hybrid cloud leverages these requirements, because it´s necessary to evaluate the resources in private cloud and, if necessary, obtain and prepare on-demand resources in the public cloud. Moreover, the simultaneous management of private and public domains requires an appropriate model that combines performance with minimal cost. In this paper we propose an architecture to make the orchestration of Hadoop applications in hybrid clouds. The core of the model consists of a web portal for submissions, an orchestration engine and an execution services factory. Through these three components it´s possible to automate the preparation of a cross-domain cluster, performing the provisioning of files involved, managing the execution of the application, and making the results available to the user.