MVPA Permutation Schemes: Permutation Testing in the Land of Cross-Validation

Permutation tests are widely used for significance testing in classification-based fMRI analyses, but the precise manner of relabeling varies, and is generally non-trivial for MVPA because of the complex data structure. Here, we describe two common means of carrying out permutation tests. In the first, which we call the "dataset-wise" scheme, the examples are relabeled prior to conducting the cross-validation, while in the second, the "fold-wise" scheme, each fold of the cross-validation is relabeled independently. While the dataset-wise scheme maintains more of the true dataset\´s structure, additional work is needed to determine which method should be preferred in practice, since the two methods often result in different null distributions (and so p-values).