A methodology for information theoretic feature extraction

We discuss an unsupervised feature extraction method which is driven by an information theoretic based criterion: mutual information. While information theoretic signal processing has been examined by many authors the method presented here is more closely related to the approaches of Linsker (1988, 1990), Bell and Sejnowski (1995), and Viola et al. (1996). The method we discuss differs from previous work in several aspects. It is extensible to a feed-forward multilayer perceptron with an arbitrary number of layers. No assumptions are made about the underlying PDF of the input space. It exploits a property of entropy coupled with a saturating nonlinearity resulting in a method for entropy manipulation with computational complexity proportional to the number of data samples squared This represents a significant computational savings over previous methods. As mutual information is a function of two entropy terms, the method for entropy manipulation can be directly applied to the mutual information as well