On the selection of testbeds for the evaluation of sensor network protocols and applications

Wireless sensor network protocols and applications typically need to be evaluated and tested not only using simulators but also on testbeds. While simulations allow studying the performance of protocols and applications in a controlled environment, they usually do not provide a sufficient level of realism. On the contrary, testbeds allow exposing protocols and applications to the real vagaries of wireless communication as well as to other “non-idealities” that typically occur in real wireless sensor network scenarios. Experimental results gathered in one or more testbeds are thus typically reported in most recent research papers on wireless sensor networks. Usually, the higher the number of different testbeds has been used, the more significant the obtained results are considered to be. However, it is often unclear whether the used testbeds actually expose protocols and applications to significantly different experimental conditions. Experiments involving a high number of testbeds are very time-consuming and cumbersome to run, but cannot guarantee that a protocol or application has been evaluated or tested in significantly different scenarios. In this paper, we argue that a systematic methodology that allows describing how significant the differences between testbeds actually are is needed. As a first step towards the definition of this methodology, we define several quantitative properties that can be used to describe and compare testbeds in a coherent manner. We show how a representative subset of these properties can be computed in real testbeds and present the results obtained by running this computation on two different testbeds. Furthermore, we describe how we plan to use our methodology to allow researchers to automatically select - out of a set of testbeds - those that are most adequate to run a specific experiment.