Suggesting Songs for Media Creation Using Semantics

In this paper, we describe a method for matching song lyrics with semantic annotations of picture collections in order to suggest songs that reflect picture content in lyrics or genre. Picture collections are first analyzed to extract a variety of semantic information including scene type, event type, and geospatial information. When aggregated over a picture collection, this semantic information forms a semantic signature of the collection. Typical picture collections in our scenario consist of photo subdirectories in which people store pictures of a place, activity, or event. Picture collections are expected to contain coherent semantic content describing in part or whole the event or activity they depict. The semantic signature of a picture collection is compared against song lyrics using a WordNet expansion based text matching to find songs relevant to the collection. We present interesting song suggestions, compare and contrast scenarios with human versus machine labels, and perform a user study to validate the usefulness of the proposed method. The proposed method will be a useful tool to support user media creation.