DZMQ: A Decentralized Distributed Messaging System for Realtime Web Applications and Services

Message-oriented middleware especially for message queue has been widely used in web applications and services. Performance and scalability are quite essential in these systems however they often become the bottleneck. Existing message queues are not able to scale out elastically very well. This paper presents a decentralized distributed architecture based on peer to peer model, in which we always deliver messages with zero or one hop and take advantage of zero-copy. We implemented a scaling algorithm that can be adapted to the dynamic scale of requests and make the system scale out elastically. A series of workload tests have proved that our system can have low response latency and achieve linear increasing throughput. With these desired properties, the message system can be used to develop large scale web applications and services and provide high-performance services to users.