A system for semantic information extraction from mixed soundtracks deploying MARSYAS framework

Ever increasing volumes of media content and the desire to extract information from media archives motivate the studies into semantic audio information mining. Much research in this filed concerns development of bespoke systems, in which sound tracks are exclusively classified and segmented, and a specific type of sound is recognized and analyzed. This approach however is detrimental to the complete extraction of all relevant semantic information and audio scene analysis. The current study addresses the issues of sound tracks with overlapped music, speech and ambient sounds, and explores how MARSYAS (Music Analysis, Retrieval and Synthesis for Audio Signals) can be extended to mixed and overlapped soundtrack applications. The MARSYAS has been adapted to this application by means of adopting additional speech cleaning algorithms. The proposed new system can analyze arbitrary sound tracks and timestamp the occurrence of music and speech, allowing overlaps, in the form of a “sound score” for further recognition methods to extract music score and text information. Validation tests have shown that the new system handles overlapping cases and is therefore capable of extracting more information than other existing methods.