Fast efficient and scalable Core Consistency Diagnostic for the parafac decomposition for big sparse tensors

Multilinear analysis is pervasive in a wide variety of fields, ranging from Signal Processing to Chemometrics, and from Machine Vision to Data Mining. Determining the quality of a given tensor decomposition is a task of utmost importance that spans all fields of application of tensors. This task by itself is hard in its nature, since even determining the rank of a tensor is an NP-hard problem. Fortunately, there exist heuristics in the literature that can be effectively used for this task; one of these heuristics is the so-called Core Consistency Diagnostic (CORCONDIA) which is very intuitive and simple. However simple, computation of this diagnostic proves to be a very daunting task even for data of medium scale, let alone big tensor data. With the increase of the size of the tensor data that need to be analyzed there grows the need for efficient and scalable algorithms to compute diagnostics such as CORCONDIA, in order to assess the modelling quality. In this work we derive a fast and exact algorithm for CORCONDIA which exploits data sparsity and scales very well as the tensor size increases.