Optimizing Support Vector regression hyperparameters based on cross-validation

This paper proposes a method to optimize hyperparameters for Support Vector (SV) regression so that the cross-validation error is minimized. The performance of SV regression depends on its hyperparameters such as ε (the thickness of a tube), C (a penalty factor), σ (kernel function parameter), and so on. This paper employs the procedure of cross-validation to optimize these hyperparameters together with training the corresponding SV regression models; thus, the learning is performed by using a coordinate descent method. Since an error surface produced by the usual ε-insensitive l₁ loss is not smooth, not suitable for our approach, we introduce the ε-insensitive l₂ loss. The experiments show the l₂ loss produces very smooth error surfaces and our coordinate descent nicely works, reaching the model whose validation performance is globally optimal.