Automatic Image Annotation Combining the Content and the Context of Medical Images

In this paper we evaluate the relevance of the information extracted from the visual content of medical images and from the image-related text-regions, as well as the performance gain obtained by combining the two approaches. First we annotate the images using a content-based annotation method that relies on the supervised classification of reduced visual representations derived from statistic and texture features. The context of medical images (i.e. image-related text regions) is extracted from the documents and analyzed using the MeSH medical ontology, which we improved and adapted to be able to extract specific category terms. Several ways of combining the two approaches are proposed and tested, showing significant overall annotation improvements.