Entropy for singular distributions

Entropy and differential entropy are important quantities in information theory. A tractable extension to singular random variables (which are neither discrete nor continuous) has not been available so far. Here, we propose such an extension for the practically relevant class of singular probability measures that are supported on a lower-dimensional subset of Euclidean space. We show that our entropy transforms in a natural manner under Lipschitz functions and that it conveys useful expressions of the mutual information. Potential applications of the proposed entropy definition include capacity calculations for the vector interference channel, compressed sensing in a probabilistic setting, and capacity bounds for block-fading channel models.