Pepper: An Elastic Web Server Farm for Cloud Based on Hadoop

Web application based processing is traditionally used to handle high throughput traffic. Web applications are hosted on server farms. However, providing application level scalability and isolation on such server farms has been a challenge. Using cloud-serving infrastructures instead could potentially provide advantages such as scalability, centralized deployment and capacity planning. They also possess attractive qualities such as self-healing as well as ease in isolation and monitoring. The problem with this approach lies in the complicated nature and operational overhead of bootstrapping and operating cloud virtualization infrastructure. We present Pepper, a novel, simple, low cost and elastic web serving cloud platform built leveraging Hadoop and Zookeeper. The design of Pepper demonstrates its ability to run in isolation different web applications and scale dynamically on a cluster of machines. Pepper is being successfully used in Yahoo! to run web applications that acquire and pre-process high frequency web feeds such as breaking news and finance quotes. Pepper processes feeds with low latency with the ability to scale to millions of feeds every day that enables us to retain content freshness.