A fast automatic construction algorithm for kernel fisher discriminant classifiers

Nonlinear Fisher Discriminant Analysis for binomial problems can be converted into a Linear-In-The-Parameters classifier model by introducing a least-squares cost function. However, the complexity of the classifier scales with the number of training samples, which makes it difficult to use on large data sets. A popular solution is to adopt a sub-model selection approach, such as Orthogonal Least Squares (OLS) or the Fast Recursive Algorithm (FRA), to produce a compact classifier with accurate parameters. The problem is that these methods need additional subjective choice of selection termination criterion, and inappropriate choice of this criterion may lead to an over-fitting classifier. Further, training data with large noise may even deteriorate the performance. This paper proposes a fast automatic forward algorithm for constructing a parsimonious descriptor of the nonlinear discriminant function, thus both the subjective choice of the termination criterion and the over-fitting problem due to noisy data can be avoided. This is achieved by an effective integration of the Bayesian regularisation technique, the Leave-One-Out (LOO) cross-validation criterion and the FRA algorithm. Experimental results are included to confirm the efficacy and superiority of the proposed algorithm on both artificial and real world data sets.