An architecture for incremental information fusion of cross-modal representations

We present an architecture for natural language processing that parses an input sentence incrementally and merges information about its structure with a representation of visual input, thereby changing the results of parsing. At each step of incremental processing, the elements in the context representation are judged whether they match the content of the sentence fragment up to that step. The information contained in the best matching subset then influences the result of parsing the subsentence. As processing progresses and the sentence is extended by adding new words, new information is searched in the context to concur with the expanded language input. This incremental approach to information fusion is highly adaptable with regard to the integration of dynamic knowledge extracted from a constantly changing environment.