Optimally regularised kernel Fisher discriminant analysis

Mika et al. (1999) introduce a non-linear formulation of Fisher\´s linear discriminant, based the now familiar "kernel trick", demonstrating state-of-the-art performance on a wide range of real-world benchmark datasets. In this paper, we show that the usual regularisation parameter can be adjusted so as to minimise the leave-one-out cross-validation error with a computational complexity of only O(ℓ²) operations, where ℓ is the number of training patterns, rather than the O(ℓ⁴) operations required for a naive implementation of the leave-one-out procedure. This procedure is then used to form a component of an efficient hierarchical model selection strategy where the regularisation parameter is optimised within the inner loop while the kernel parameters are optimised in the outer loop.