Title :
Predicting User Satisfaction in Spoken Dialog System Evaluation With Collaborative Filtering
Author :
Zhaojun Yang ; Levow, G.-A. ; Meng, Hsiang-Yun
Author_Institution :
Dept. of Electr. Eng., Univ. of Southern California, Los Angeles, CA, USA
Abstract :
We propose a collaborative filtering (CF) model to predict user satisfaction in SDS evaluation. Inspired by the use of CF in recommendation systems, where a user´s preference for a new item is assume to resemble that for similar items rated previously, we adapt the idea to predict user evaluations of unrated dialogs based on the ratings received by similar dialogs. Ratings of dialogs are gathered by crowdsourcing through Amazon Mechanical Turk. A reference baseline is provided by a linear regression model (LRM) based on the PARADISE framework. We present two versions of the CF model. First, the item-based collaborative filtering model (ICFM) clusters rated dialogs and builds an LRM for each cluster. The rating of an unseen dialog is predicted by the LRM of its most similar cluster. Second, the extended ICFM (EICFM) separates dialog features into user-related and system-related groups, to build LRMs for these separately. Experimental results on dialogs from the Let´s Go! system show both ICFM and EICFM can significantly improve the proportion of variability explained by the LRM. We also demonstrate the generalizability of the CF model to a new dialog corpus from the systems in the Spoken Dialog Challenge (SDC) 2010.
Keywords :
collaborative filtering; human computer interaction; interactive systems; outsourcing; pattern clustering; recommender systems; regression analysis; speech recognition; Amazon Mechanical Turk; CF model; EICFM; ICFM clusters rated dialogs; Let´s Go! system; PARADISE framework-based LRM; PARADISE framework-based linear regression model; SDC 2010; SDS evaluation; Spoken Dialog Challenge 2010; collaborative filtering; crowdsourcing; dialog corpus; dialog ratings; extended ICFM; item-based collaborative filtering model clusters rated dialogs; most similar cluster; recommendation systems; reference baseline; spoken dialog system evaluation; system-related groups; user satisfaction prediction; user-related groups; Accuracy; Filtering; Human computer interaction; Linear regression; Predictive models; Speech recognition; Spoken dialog system evaluation; collaborative filtering; crowdsourcing user satisfaction;
Journal_Title :
Selected Topics in Signal Processing, IEEE Journal of
DOI :
10.1109/JSTSP.2012.2229965