MVPA Permutation Schemes: Permutation Testing for the Group Level

Permutation tests are widely used for significance testing in fMRI MVPA (multivariate pattern analysis) studies, but the precise way in which the tests are carried out varies, and test design is non-trivial because of complex, auto correlated, and stratified dataset structures. Previously, we described permutation tests for single-subject datasets, recommending adoption of "dataset-wise" schemes, in which examples are relabeled prior to cross-validation. Here, we extend that work by describing permutation schemes for group analyses: datasets with more than one participant. Group-level MVPA is most often performed with either cross-validation on the subjects or within-subjects cross-validation, each of which requires a different strategy for permutation testing, as illustrated here.