What can visual content analysis do for text based image search?

Modern image search engines such as Google, Yahoo!, Microsoft Live image search are all text meta word based. To search for images, the users type in a text query and the search engines rank the result images almost sorely based on the text meta-words. The abundant visual information in the images themselves is largely neglected. Recently, we have observed several new features released in the aforementioned image search engines, especially Microsoft Live image search, which are clearly based on the analysis of the visual content. We summarize some of these features, give insights about how they are designed, and motivate new content analysis based features for text based image search engines.