Towards correct and informative evaluation methodology for texture classification under varying viewpoint and illumination

3D texture classification under varying viewpoint and illumination has been a vivid research topic, and many methods have been developed. It is crucial that these methods be compared using an unbiased evaluation methodology. The most frequently employed methodologies use images from the Columbia–Utrecht Reflectance and Texture Database. These methodologies construct the training and test sets to be disjoint in the imaging parameters, but do not separate them spatially because they use images of the same surface patch for both. We perform a series of experiments which show that such practice leads to overestimation of classifier performance and distorts experimental findings. To correct that, we accurately register the images across all imaging conditions and split the surface patches to parts. The training and testing is then done on spatially disjoint parts. We show that such methodology gives a more realistic assessment of classifier performance. The sample annotations for all images are publicly available.