Bootstrapping word learning: A perception driven semantics-first approach

In recent decades, evidence for preverbal perceptual categorization in infants has been accumulating, and a role has been suggested for such processes in bootstrapping word learning, i.e. acquiring the very first word-meaning associations. We propose a computational study to consider the possibility that initial notions of semantic classes may help word learning. We consider visual scenes with many possible referents, and consider unparsed linguistic descriptions in text form. We use no prior knowledge of vision domain, or of morphology, syntax or word frequency. Using a synthetic model of object attention, we show that the system is able to first identify perceptual classes from the visual stream, and then associate these with words from the linguistic stream. Working with Hindi text, we demonstrate the ability to learn words for prominent proto-concepts like BICYCLE, TRUCK, and CAR from a complex traffic video. We compare the associations when learning unsegmented poly-syllabic strings in the language (without knowledge of word boundaries) versus segmented words, and find that the poly-syllables do nearly as well. This suggests that early acquisition of some semantic classes may also help in parsing the input stream into “words”. The model is then used on a novel video from a similar domain, to identify objects with their labels. Since we provide no knowledge to the system either for the visual or language analyses, the results are likely to hold for other visual scenes and languages.