Variational Bayesian Learning of Probabilistic Discriminative Models With Latent Softmax Variables

This paper presents new variational Bayes (VB) approximations for learning probabilistic discriminative models with latent softmax variables, such as subclass-based multimodal softmax and mixture of experts models. The VB approximations derived here lead to closed-form approximate parameter posteriors and suitable metrics for model selection. Unlike other Bayesian methods for this challenging class of models, the proposed VB methods require neither restrictive structural assumptions nor sampling approximations to cope with the problematic softmax function. As such, the proposed VB methods are also easily extendable to more complex softmax-based hierarchical discriminative models and regression models (for continuous outputs). The proposed VB methods are evaluated on benchmark classification data and a decision modeling application, demonstrating good results.