Correcting rolling-shutter distortion of CMOS sensors using facial feature detection

This paper proposes a fully automated post image processing scheme based on facial feature detection to correct the horizontal temporal shear or rolling shutter distortion. This distortion occurs when obtaining images or video sequences from a CMOS camera with a rolling shutter whenever there is relative horizontal movement between the sensor and the object being imaged during the integration time of the image frame. Unlike CCD sensors, such as the interline CCD, which provides an electronic shutter mechanism called a global shutter in which the light collection starts and ends at exactly the same time for all pixels, CMOS sensors can not hold and store all the pixels at the same time. Each scanline is exposed, sampled, and stored in sequence, resulting in the rolling shutter effect or temporal distortion of the image that will cause inaccurate facial recognition results. Facial feature detection is performed using correlation based methods with low computational complexity. The location of key facial feature points is then used to calculate the temporal horizontal shear or the distortion of the image. This information can then be used to remove the temporal horizontal shear distortion from the detected face or the entire image. We present experimental results on controlled data sets and real scenes to show that the proposed method yields excellent results in reversing the temporal horizontal shear caused by the CMOS rolling shutter sensor and significantly improves the accuracy of our facial recognition algorithm.