Autonomic Management of Stream Processing Applications via Adaptive Bandwidth Control

We present a novel autonomic control system for high performance stream processing systems. The system uses bandwidth controls on incoming or outgoing streams to achieve a desired resource utilization balance among a set of concurrently executing stream processing tasks. We show that CPU prioritization and allocation mechanisms in schedulers and virtual machine managers are not sufficient to control such I/O-centric applications, and present an autonomic bandwidth control system that adaptively adjusts incoming and outgoing traffic rates to achieve system management goals. The system dynamically learns the bandwidth rate necessary to meet the system management goals using stochastic nonlinear optimization, and detects changes in the stream processing applications that require bandwidth adjustment. Our prototype Linux implementation is lightweight, has low overhead, and is capable of effectively managing stream processing applications.