Adding stream processing system flexibility to exploit low-overhead communication systems

Previously, we demonstrated that we can build a real-world financial application using a stream processing system running on commodity hardware. In this paper, we propose making stream processing systems more flexible and demonstrate how this flexibility can be used to exploit low-overhead communication systems to speed up streaming applications. With our prototype, we now have an options market data processing system that can achieve less than 30 µsec average latency at 30x the February 2008 OPRA rate on a cluster of blades using InfiniBand. Across shared memory, this system can achieve less than 20 µsec average latency at 25x the February 2008 OPRA rate on a single machine.