Annotating Unconstrained Face Imagery: A scalable approach

As unconstrained face recognition datasets progress from containing faces that can be automatically detected by commodity face detectors to face imagery with full pose variations that must instead be manually localized, a significant amount of annotation effort is required for developing benchmark datasets. In this work we describe a systematic approach for annotating fully unconstrained face imagery using crowdsourced labor. For such data preparation, a cascade of crowdsourced tasks are performed, which begins with bounding box annotations on all faces contained in images and videos, followed by identification of the labelled person of interest in such imagery, and, finally, landmark annotation of key facial fiducial points. In order to allow such annotations to scale to large volumes of imagery, a software system architecture is provided which achieves a sustained rate of 30,000 annotations per hour (or 500 manual annotations per minute). While previous crowdsourcing guidance described in the literature generally involved multiple choice questions or text input, our tasks required annotators to provide geometric primitives (rectangles and points) in images. As such, algorithms are provided for combining multiple annotations of an image into a single result, and automatically measuring the quality of a given annotation. Finally, other guidance is provided for improving the accuracy and scalability of crowdsourced image annotation for face detection and recognition.