Graph-Based Partial Hypothesis Fusion for Pen-Aided Speech Input

We study a specific partial hypothesis fusion problem in sequential data labeling. The problem arises in the multimodal applications where a decision is made by merging complete hypothesis from one input and partial hypothesis from the other. For example, in a pen-aided speech interface, appropriate pen input can provide partial but crucial information. We address the problem in a Bayesian framework, and reformulate the solution as a revised search in a representation. A dynamic programming algorithm is proposed to efficiently solve the partial hypothesis fusion via the graph. It is shown that the computational cost of the graph based partial hypothesis fusion is proportional to the size of the graph, which is highly feasible for a given compact graph. We apply the proposed algorithm to two real applications: an intelligent pen-based dictation error correction system and an automatic handwritten character completion with a speech ldquoshortcutrdquo. Experimental results show that the algorithm is effective in utilizing the partial information from one modality to enhance the bimodal interface performance.