Kernel second-order discriminants versus support vector machines

Support vector machines (SVMs) are the most well known nonlinear classifiers based on the Mercer kernel trick. They generally lead to very sparse solutions that ensure good generalization performance. Recently, S. Mika et al. (see Advances in Neural Networks for Signal Processing, p.41-8, 1999) proposed a new nonlinear technique based on the kernel trick and the Fisher criterion: the nonlinear kernel Fisher discriminant (KFD). Experiments show that KFD is competitive with SVM classifiers. Nevertheless, it can be shown that there exist distributions such that even though the two classes are linearly separable, the Fisher linear discriminant has an error probability close to 1. We propose an alternative strategy based on Mercer kernels that consists in picking the optimum nonlinear receiver in the sense of the best second-order criterion. We also present a strategy for controlling the complexity of the resulting classifier. Finally, we compare this new method with SVM and KFD.