Granting space and time to words: Blind semantic spatial localization for the case of facial images

There exist a multitude of multimedia sources nowadays where audiovisual material is accompanied by labels. In many cases, the semantic content of these labels is not referring to the totality of the material they accompany; but only to a certain subset of it. Consider, for example, labels that refer to objects that are part of an image; or, alternatively, labels that refer to an event that is part of a video. Knowledge of the subset of the material to which the labels refer to can be very useful; for example, it can inform us regarding the detectability of the entity under the presence of selective occlusions or noise; or, it can be used to help segment out the referent from the material itself - and, among many other uses, to optimize the recognition of the entities that the words refer to. Towards these goals, in this paper we will present a method which allows such semantic spatiotemporal localization: given multiple instances of the material, and accompanying labels, we will produce subsets of the material which are most informative regarding the label; and which can be thought of as the spatiotemporally localized grounding of the concept represented by the words. The method is illustrated for the specific case of spatially localizing labels describing human faces or parts and artifacts of them; such as “beard”, “glasses”, “male”, “old”. No prior information about the spatial locus of the referents of these words is given; the algorithm blindly identifies the regions that are most informative for each label, and can be readily applied to robot vision.