WebRTC quality assessment: Dangers of black-box testing

We expect that WebRTC will experience high adoption rate as a peer-to-peer realtime communication standard for browsers. WebRTC allows direct media and data transport between browsers without having to go through a web server. In this study, we use a black-box testing technique to evaluate, via PESQ, the voice quality of WebRTC sessions under varying network delay and jitter. Network emulators are employed to implement the delay and jitter variations. Our results highlight the dangers of black-box testing, whereby test-bed issues can result in very misleading results. This is especially the case when executed on a single machine. This paper also provides an extendable baseline methodology for WebRTC centric research.