Detecting Problematic Turns in Human-Machine Interactions: Rule-induction Versus Memory-based Learning Approaches

Detecting problematic turns in human-machine interactions: Rule-induction versus memory-based learning approaches Antal van den Bosch � � ILK / Comp. Ling. KUB, Tilburg The Netherlands antalb@kub.nl Emiel Krahmer �� � � IPO TU/e, Eindhoven The Netherlands E.J.Krahmer@tue.nl Marc Swerts �� � � CNTS UIA, Antwerp Belgium M.G.J.Swerts@tue.nl Abstract We address the issue of on-line detec- tion of communication problems in spoken dialogue systems. The useful- ness is investigated of the sequence of system question types and the word graphs corresponding to the respective user utterances. By applying both rule- induction and memory-based learning techniques to data obtained with a Dutch train time-table information system, the current paper demonstrates that the aforementioned features indeed lead to a method for problem detec- tion that performs significantly above baseline. The results are interesting from a dialogue perspective since they employ features that are present in the majority of spoken dialogue systems and can be obtained with little or no computational overhead. The results are interesting from a machine learning perspective, since they show that the rule-based method performs signific- antly better than the memory-based method, because the former is better capable of representing interactions between features. 1 Introduction Given the state of the art of current language and speech technology, communication problems are unavoidable in present-day spoken dialogue sys- tems. The main source of these problems lies in the imperfections of automatic speech recogni- tion, but also incorrect interpretations by the nat- ural language understanding module or wrong de- fault assumptions by the dialogue manager are likely to lead to confusion. If a spoken dialogue system had the ability to detect communication problems on-line and with high accuracy, it might be able to correct certain errors or it could in- teract with the user to solve them. For instance, in the case of communication problems, it would be beneficial to change from a relatively natural dialogue strategy to a more constrained one in order to resolve the problems (see e.g., Litman and Pan 2000). Similarly, it has been shown that users switch to a ‘marked’, hyperarticulate speak- ing style after problems (e.g., Soltau and Waibel 1998), which itself is an important source of re- cognition errors. This might be solved by using two recognizers in parallel, one trained on nor- mal speech and one on hyperarticulate speech. If there are communication problems, then the sys- tem could decide to focus on the recognition res- ults delivered by the engine trained on hyperartic- ulate speech. For such approaches to work, however, it is essential that the spoken dialogue system is able to automatically detect communication problems with a high accuracy. In this paper, we investigate the usefulness for problem detection of the word graph and the history of system question types. These features are present in many spoken dia- logue systems and do not require additional com- putation, which makes this a very cheap method to detect problems. We shall see that on the basis of the previous and the current word graph and …

Content truncated to 500 words