Fusion of fMRI, sMRI, and EEG data using canonical correlation analysis

Typically data acquired through imaging techniques such as functional magnetic resonance imaging (fMRI), structural MRI (sMRI), and electroencephalography(EEG) are analyzed separately. Each modality records brain structure and function at different scales, and fusing information from such complementary modalities promises to provide additional insight into connectivity across brain networks and changes due to disease. Recently, a number of methods have been proposed for data integration and fusion of two brain imaging modalities. We propose a new data fusion scheme based on canonical correlation analysis that enables the detection of associations across multiple modalities. Our multimodal canonical correlation analysis (mCCA) scheme works at the feature level using multi-set CCA to determine inter-subject covariations across modalities. We apply mCCA to fMRI, sMRI, and EEG data collected from patients diagnosed with schizophrenia and healthy controls. Through data collected from an auditory oddball task, we show that the fusion of multiple modalities detects more specific associations as compared to fusion of two modalities.