Removing fillers to induce semantic classes for a Chinese dialogue system

In this paper, we introduced an unsupervised method to remove fillers in spoken dialogues semi-automatically based on their probability distribution. Disfluencies such as fillers, repairs often make the sentence ill-formed, longer and hard to process. Fillers were emphasized instead of repairs in this paper. We conduct the unigram and bigram distribution of fillers on our Chinese voice search data and find that only using these distributions, fillers are in the first 1% of all words. We give a new perspective of fillers distribution and new measure to detect fillers on the natural dialogue corpus.