Discovering the concept of anaphora from grounded verb models

A number of computational models simulate the grounded learning of units of language in the early learner. But can this initial lexical knowledge be used to acquire complex grammatical notions such as anaphora? We build on earlier work, where we simulate a language learner with perceptual attention and learn, in an unsupervised manner, a set of action models along with the participating agents, and then the corresponding linguistic units (verbs). We consider how this knowledge may be used to bootstrap the learning of anaphora. Given an input video with moving shapes, the system considers human narratives that refer to this scene. The acquired perceptual schemas and their arguments are mapped to the appropriate verbs and nouns in the discourse. We first detect the synonyms of the arguments as the repeated labels used in the constructions referring to a known action scene. After ruling out the synonyms, we find that the anaphora remain as units that are referring to more than one grounded object. We show that both third-person singular and plural anaphors and even a common reciprocal anaphor (“each other”) can be discovered. Finally, we show that in situations where the referent is missing altogether (zero anaphora), certain correlations may also be inferred from the regularities in the mapping between perceptual schemas and language.