Classifying Biomedical Figures Using Combination of Bag of Keypoints and Bag of Words

Figures in full science papers contain much information not described in the main text. The use of these figures is an important subject. In this study, we developed a method to classify these biomedical figures into categories using a combination of a bag of keypoints approach and a bag of words approach for the legends. For bag of keypoints, the descriptors of detected interest points are quantized by k-means clustering and are converted into a feature vector element. The figure classification is carried out using a one-against-all multi-class support vector machine. When the Harris-affine and extended scale-invariant feature transform method are used for interest point detection and feature description, respectively, the classification accuracy of bag of keypoints is about 20% better than that of field-level image descriptors which are similar to descriptors used in previous studies. Further, when bag of words for legends is combined with this method, the prediction performance achieved 75.7% classification accuracy. Introducing bag of keypoints not only increases the classification performance but also enables the figures to be treated as text words. This method is expected to be useful for figure similarity search and information retrieval across figures and text.