Performance Prediction Challenge

A major challenge for machine learning algorithms in real world applications is to predict their performance. We have approached this question by organizing a challenge in performance prediction for WCCI 2006. The class of problems addressed are classification problems encountered in pattern recognition (classification of images, speech recognition), medical diagnosis, marketing (customer categorization), text categorization (filtering of spam). Over 100 participants have been trying to build the best possible classifier from training data and guess their generalization error on a large unlabeled test set. The challenge scores indicate that cross-validation yields good results both for model selection and performance prediction. Alternative model selection strategies were also sometimes employed with success. The challenge web site keeps open for post-challenge submissions: http://www.modelselect.inf.ethz.ch/.