Testing Multiclass Pattern Discrimination

Machine learning is increasingly adopted in neuroimaging-based neuroscience studies. The paradigm of predicting the stimuli provided to the subject from the concurrent brain activity is known as "brain decoding" and accurate predictions support the hypothesis that the brain activity encodes those stimuli. When the stimulus categories are more than two it is not straightforward how to assess the amount of evidence in support of such an hypothesis. Moreover it is unclear how to distinguish between a classifier that discriminates each single class from the one that discriminates only among subsets of the classes. In this work we propose to recast the testing problem as a test of statistical independence between the predicted and the actual class labels. In this setting we propose a novel method to test whether the classifier is able to discriminate all classes or just subsets of them. We show experimental evidence of its efficacy both on simulated and on real data from an MEG experiment.