Learning feature transforms is an easier problem than feature selection

We argue that optimal feature selection is intrinsically a harder problem than learning discriminative feature transforms, provided a suitable criterion for the latter. We discuss mutual information between class labels and transformed features as such a criterion. Instead of Shannon\´s definition we use measures based on Renyi entropy, which lends itself into an efficient implementation and an interpretation of "information forces" induced by samples of data that drive the transform.