When brain and behavior disagree: Tackling systematic label noise in EEG data with machine learning

Conventionally, neuroscientific data is analyzed based on the behavioral response of the participant. This approach assumes that behavioral errors of participants are in line with the neural processing. However, this may not be the case, in particular in experiments with time pressure or studies investigating the threshold of perception. In these cases, the error distribution deviates from uniformity due to the heteroscedastic nature of the underlying experimental set-up. This problem of systematic and structured (non-uniform) label noise is ignored when analysis are based on behavioral data, as is being done typically. Thus, we run the risk to arrive at wrong conclusions in our analysis. This paper proposes a remedy to handle this crucial problem: we present a novel approach for a) measuring label noise and b) removing structured label noise. We show its usefulness for an EEG data set recorded during a standard d2 test for visual attention.