Multimodal interfaces with voice and gesture input

The modalities of speech and gesture have different strengths and weaknesses, but combined they create a synergy where each modality corrects the weaknesses of the other. The authors believe that a multimodal system such as one intertwining speech and gesture must start from a different foundation than ones which are based solely on pen input. In order to provide a basis for the design of a speech and gesture system, the authors have examined the research in other disciplines such as anthropology and linguistics. The result of this investigation was a taxonomy that gave the authors material for the incorporation of gestures whose meanings are largely transparent to the users. This study describes the taxonomy and gives examples of applications to pen input systems