Information theoretic measures for performance evaluation and comparison

This paper discusses the performance comparison of different algorithms for classification, estimation and filtering problems. Two information theoretic measures, namely, the empirical mutual information and the asymptotic information rate are proposed for simulation based performance evaluation and algorithm comparison. They can be used as a guideline for designing a practical procedure to measure the performance of different algorithms with limited computational resources. Other useful performance measures are reviewed and their relation to the two new measures discussed. Several practical examples are used to provide some insights on the inherent difficulty of algorithm ranking and the advantage of using the information theoretic measures for algorithm comparison.