A framework for evaluation of middleware systems of mobile multimedia services

In this paper, we present a framework that we have developed for evaluating middleware systems of mobile multimedia services. In this framework, we combine subjective evaluation criteria relating to users´ responses and objective evaluation criteria relating to software parameters. In order to implement this criteria combination, we developed an evaluation framework consisting of three stages. In the first stage, we conduct an empirical study to evaluate various aspects of the usefulness of the mobile multimedia service. In the second stage, we collect qualitative and quantitative software parameters of the middleware system. Finally, in the third stage we examine the overall assessment of the system. For illustration purposes, the evaluation framework is applied to a middleware system that we have developed to facilitate mobile access to digital music libraries.