Real-time heads-up display detection in video

Video from surveillance cameras, aerial sensors, video games, and other sources may occasionally contain text, heads-up displays (HUDs), lens debris, or other artifacts superimposed on top of some scene. In standard video processing pipelines, the early detection and filtering of these image-plane aligned obstructions can be helpful for improving the accuracy of later operations, such as video stabilization, tracking, or object recognition. This paper presents one such technique to automatically accomplish this, which first extracts various pixel-level features which jointly take into account local spatiotemporal variations around each pixel. Features extracted from multiple frames are then utilized by a novel classification system to determine if any obstructions are present and, if possible, to categorize them into known types. Experimental results show promising performance on a variety of different categories of HUD, in addition to other types of on-screen display.