Using Semantics for Speech Annotation of Images

In this paper, we have postulated the problem of using discrete speech utterances to annotate an image as that of disambiguation across multiple N-best lists. Our solution is based on the Maximum Entropy approach and uses correlations between tags in an existing corpus of images to set up the constrains of the corresponding constrained optimization problem. Our experiments suggest that the proposed approach gives a significant improvement in quality as compared to an approach that considers the best answer suggested by a popular off-the-shelf recognizer.