Scaling up applications over distributed clouds with dynamic layer-2 exchange and broadcast service

In this paper, we study the problem of provisioning large-scale virtual clusters over federated clouds connected by multi-domain, layer-2 wide area networks. We first present the virtual cluster request abstraction and the abstraction models for substrate resource pools. Based on these two abstraction models, we developed a novel layer-2 exchange mechanism and an implementation of it in a multi-domain networked cloud environment. The design of the mechanism takes into consideration the realistic constraints in current network and cloud systems. We show that efficient cluster splitting, cloud data center selection and resource allocation algorithms can be developed to provision large-scale virtual clusters across cloud sites. A prototype system has been deployed and integrated into the ExoGENI testbed for about a year, and is being heavily used by scientific and data analytic applications.