Information-Theoretic Variable Selection in Neural Networks

In this paper, we propose a new type of information-theoretic approach to variable selection. Many approaches have been proposed in estimating the importance of input variables. The majority of these approaches have focused upon output errors. We here introduce an approach concerning internal representations. First, we delete an input unit with corresponding connection weights. Then, by examining some change in hidden unit activation with and without a input variable, we can extract an important variable. We apply this method to an artificial data in which the number of hidden units is redundantly increased so as to clearly show improved performance and the stability of our method. Then, we apply the method to the cabinet approval ratings in which better interpretation of input variables can be given