Learning from multimodal observations

Human-computer interaction based on recognition of speech, gestures, and other natural modalities is on the rise. Recognition technologies are typically developed in a statistical framework and require large amounts of training data. The cost of collecting manually annotated data is usually the bottleneck in developing such systems. We explore the idea of learning from unannotated data by leveraging information across multiple modes of input. A working system inspired by infant language learning which learns from untranscribed speech and images is presented