A novel similarity measure for semantic class induction in human-computer spoken dialogues

In this paper, we introduced a new semantic induction metric which can induce some semantic classes from a set of domain-specific unannotated data. We emphasized on the co-occurrence probability instead of just distances of word probability distribution. Compared to the traditional approach on right or left context to calculate the similarity, we used both left and right information simultaneously in the metric. Before processing, we removed fillers based on their unigram and bigram context distribution. We can find that co-occurrence metric is simple, effective and have lower misclassified portion. We test the metric on our Chinese voice-search data, and get F1 for 84.3.