Locally linear independent component analysis

Linear independent component analysis (ICA) has become an important technique in unsupervised neural learning. Even though linear ICA yields meaningful results in many cases, it can provide a crude approximation only for general nonlinear data distributions. We study techniques where local ICA models are applied to data first grouped or clustered using some suitable algorithms. The grouping part is responsible for an overall coarse nonlinear representation of the underlying data, while linear ICA models of each group (cluster) are used for describing local features of the data. The goal is to represent the data better than in linear ICA in a computationally feasible manner. We study several grouping methods, including standard K-means clustering, self-organizing maps, and neural gas. We also present a general theoretical framework encompassing a large number of methods for representing data. These range from global, dense representation methods to local, very sparse coding methods. The proposed local ICA methods lie between these two extremes.