Dimensionality reduction for bio-medical spectra

The classification problem for high dimensional data (for example near infrared spectra of bio-fluids) is a challenging, cornerstone problem in bio-informatics. The problems in the field possess many measured, highly correlated variables, which typically come from digitization of continuous signals, and relatively few distinct samples, with the number of variables often far exceeding the number of observations. Fortunately, in practice, the data are often restricted or nearly restricted to a relatively low dimensional manifold in feature space. We will compare several techniques both linear and nonlinear for identifying this manifold, including local and global principal component analysis, and a novel implementation of the (nonlinear) Whitney reduction network. The intrinsic dimension of the data manifold will be verified through an independent validation set.