User Simulations for Context-Sensitive Speech Recognition in Spoken Dialogue Systems

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 505–513, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics User Simulations for context-sensitive speech recognition in Spoken Dialogue Systems Oliver Lemon Edinburgh University olemon@inf.ed.ac.uk Ioannis Konstas University of Glasgow konstas@dcs.gla.ac.uk Abstract We use a machine learner trained on a combination of acoustic and contextual features to predict the accuracy of incom- ing n-best automatic speech recognition (ASR) hypotheses to a spoken dialogue system (SDS). Our novel approach is to use a simple statistical User Simulation (US) for this task, which measures the likelihood that the user would say each hypothesis in the current context. Such US models are now common in machine learning approaches to SDS, are trained on real dialogue data, and are related to the- ories of “alignment” in psycholinguistics. We use a US to predict the user’s next dia- logue move and thereby re-rank n-best hy- potheses of a speech recognizer for a cor- pus of 2564 user utterances. The method achieved a significant relative reduction of Word Error Rate (WER) of 5% (this is 44% of the possible WER improvement on this data), and 62% of the possible se- mantic improvement (Dialogue Move Ac- curacy), compared to the baseline policy of selecting the topmost ASR hypothesis. The majority of the improvement is at- tributable to the User Simulation feature, as shown by Information Gain analysis. 1 Introduction A crucial problem in the design of spoken dia- logue systems (SDS) is to decide for incoming recognition hypotheses whether a system should accept (consider correctly recognized), reject (as- sume misrecognition), or ignore (classify as noise or speech not directed to the system) them. Obviously, incorrect decisions at this point can have serious negative effects on system usability and user satisfaction. On the one hand, accept- ing misrecognized hypotheses leads to misunder- standings and unintended system behaviors which are usually difficult to recover from. On the other hand, users might get frustrated with a system that behaves too cautiously and rejects or ignores too many utterances. Thus an important feature in di- alogue system engineering is the tradeoff between avoiding task failure (due to misrecognitions) and promoting overall dialogue efficiency, flow, and naturalness. In this paper, we investigate the use of machine learning trained on a combination of acoustic fea- tures and features computed from dialogue context to predict the quality of incoming n-best recogni- tion hypotheses to a SDS. These predictions are then used to select a “best” hypothesis and to de- cide on appropriate system reactions. We evalu- ate this approach in comparison with a baseline system that works in the standard way: always choosing the topmost hypothesis in the n-best list. In such systems, complex repair strategies are re- quired when the top hypothesis is incorrect. The main novelty of this work is that we ex- plore the use of predictions from simple statisti- cal User Simulations to re-rank n-best lists of ASR hypotheses. These User Simulations are …

Content truncated to 500 words