Brain pathological image retrieval based on correlation of visual feature and report-assisted text

In the medical domain, digital images are produced in ever-increasing quantities and used for diagnostics and therapy. However, it is not sufficient to capture the semantic content of an image and difficult to provide good medical image retrieval results according to the predefined categories in the medical domain for less using the medical knowledge. This paper proposes a semantic approach by focusing on the medical image and report fusion. Firstly the feature vectors of color and texture are extracted by analyzing the image low-level features. Secondly, a set of disjoint semantic tokens with appearance in medical images is selected to define a visual and medical vocabulary. Here we use the diagnosis report from the doctor to represent each token in the medical domain. Finally the low-level features are used as input vector of SVM to training. And the semantic correlation from low-level features to high-level features is constructed according to the Visual and Medical vocabulary. This semantic correlation method has been used in retrieval experiment, which concerns the brain CT images. And the result demonstrates the improvement, effectiveness, and efficiency achieved by the proposed framework.