Using Ensembles of Neural Networks to Improve Automatic Relevance Determination

Automatic relevance determination (ARD) is an efficient technique to infer the relevance of input features with respect to their ability to predict the target output for a task. ARD optimizes the hyperparameters to maximize the evidence. This optimization can cause some hyperparameters of relevant features tends towards infinity and therefore these features are inferred as irrelevant by an ARD model. The overfitting of relevance parameters cause feature relevance determinations to be not stable and reliable. Neural network ensemble methods can utilize the diversity between ensemble members to reduce the uncertainty in order to generate a more reliable determination of input feature relevancies. Input features were properly grouped based on their relevance level by ensemble relevance prediction.