A kernel based approach to maximum entropy mappings

We discuss a kernel based method for learning maximum entropy mappings from exemplars. Information theoretic signal processing has been examined by many authors. The method presented is related to the approaches of Linsker (1988, 1990), Bell and Sejnowski (see Neural Computation, vol.7, p.1129-59, 1995), and Viola et al. (see Neural Information Processing Systems, vol.8, p.851-7, 1995). We discuss the use of this method for deriving maximum entropy mappings in an unsupervised fashion. Extensions to optimizing mutual information are possible. The result of our approach is that maximizing and minimizing entropy for differentiable nonlinear mappings such as a multilayer perceptron can be accomplished through simple local interactions of the data in the output space. We present empirical results from application of the method to the problem of blind separation of linearly mixed speech sources. We compare our empirical results to the method of Bell and Sejnowski