Title :
On the Discontinuity of the Shannon Information Measures
Author :
Ho, Siu-Wai ; Yeung, Raymond W.
Abstract :
The Shannon information measures are well known to be continuous functions of the probability distribution for a given finite alphabet. In this paper, however, we show that these measures are discontinuous with respect to almost all commonly used "distance" measures when the alphabet is countably infinite. Such "distance" measures include the Kullback-Leibler divergence and the variational distance. Specifically, we show that all the Shannon information measures are in fact discontinuous at all probability distributions. The proofs are based on a probability distribution which can be realized by a discrete-time Markov chain with countably infinite number of states. Our findings reveal that the limiting probability distribution may not fully characterize the asymptotic behavior of a Markov chain. These results explain why certain existing information-theoretical tools are restricted to finite alphabets, and provide hints on how these tools can be extended to countably infinite alphabet.
Keywords :
Markov processes; information theory; statistical distributions; Kullback-Leibler divergence; Shannon information measures; asymptotic behavior; continuous functions; discrete-time Markov chain; distance measures; probability distribution; variational distance; Entropy; H infinity control; Information theory; Probability distribution; Continuity; Shannon information measure; countable alphabet; countably infinite alphabet; discontinuity; divergence;
Journal_Title :
Information Theory, IEEE Transactions on
DOI :
10.1109/TIT.2009.2032707