Probabilistic nearest neighbor search for robust classification of face image sets

Classification with image sets is recently a compelling technique for video-based face recognition. Previous methods in this line mostly assume each image set is pure, i.e., containing well-aligned face images of the same subject, which however is hardly satisfied in real-world applications due to incorrect face detection, questionable tracking, or multiple faces in a single image. This paper proposes a Probabilistic Nearest Neighbor (ProNN) search method to enhance the robustness of NN search against impure image sets by leveraging the statistical distribution of the involved image sets. Specifically, we represent image sets by affine hull, a well-recognized set model, to account for the unseen appearances in each image set. We further exploit a constraint that these unseen appearances statistically follow some pre-specified distribution (Gaussian in this work). Finally, in search of a pair of nearest neighbor points (one per hull), at the same time their distance being minimized, the probability of each point belonging to the same class as that of its corresponding hull is maximized. The proposed ProNN method is evaluated on three widely-studied public databases, Honda/UCSD, YouTube Celebrities and Multiple Biometric Grand Challenge (MBGC), under two kinds of experimental settings where image sets are contaminated either with false positive faces or images of other subjects. Extensive experiments demonstrate the superiority of the proposed approach over state-of-the-art methods.