The Figure Understander: a tool for the integration of text and graphical input to a knowledge base

The paper describes the Figure Understander, a tool for integrating multimedia input in the form of graphically-drawn diagrams and text to a knowledge base. This tool has been designed to support applications. Such as a story or map understanding system that will benefit from the ability to input a spatial scene compactly through a diagram, but may require additional text to describe the domain properties of objects. Previous solutions to the problem of integrating text and pictorial input have been based on associating spatial images with text objects. The ambiguities associated with these methods are well known. We present an alternate approach which eliminates many of the sources of ambiguity by allowing semantic information to be provided directly within diagram objects. We illustrate the utility of our method through examples from two independent applications