Using Paraphrases and Lexical Semantics to Improve the Accuracy and the Robustness of Supervised Models in Situated Dialogue Systems

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 808–813, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Using Paraphrases and Lexical Semantics to Improve the Accuracy and the Robustness of Supervised Models in Situated Dialogue Systems Claire Gardent CNRS/LORIA, Nancy claire.gardent@loria.fr Lina M. Rojas Barahona Université de Lorraine/LORIA, Nancy lina.rojas@loria.fr Abstract This paper explores to what extent lemmati- sation, lexical resources, distributional seman- tics and paraphrases can increase the accuracy of supervised models for dialogue manage- ment. The results suggest that each of these factors can help improve performance but that the impact will vary depending on their com- bination and on the evaluation mode. 1 Introduction One strand of work in dialog research targets the rapid prototyping of virtual humans capable of con- ducting a conversation with humans in the context of a virtual world. In particular, question answering (QA) characters can respond to a restricted set of topics after training on a set of dialogs whose utter- ances are annotated with dialogue acts (Leuski and Traum, 2008). As argued in (Sagae et al., 2009), the size of the training corpus is a major factor in allowing QA characters that are both robust and accurate. In ad- dition, the training corpus should arguably be of good quality in that (i) it should contain the various ways of expressing the same content (paraphrases) and (ii) the data should not be skewed. In sum, the ideal training data should be large (more data is better data) ; balanced (similar amount of data for each class targeted by the classifier) and varied (it should encompass the largest possible number of paraphrases and synonyms for the utterances of each class). In this paper, we explore different ways of im- proving and complementing the training data of a supervised QA character. We expand the size and the quality (less skewed data) of the training corpus using paraphrase generation techniques. We com- pare the performance obtained on lemmatised vs. non lemmatised data. And we investigate how vari- ous resources (synonym dictionaries, WordNet, dis- tributional neighbours) can be used to handle unseen words at run time. 2 Related work Previous work on improving robustness of super- vised dialog systems includes detecting and han- dling out of domain utterances for generating feed- back (Lane et al., 2004) ; using domain-restricted lexical semantics (Hardy et al., 2004) ; and work on manual data expansion (DeVault et al., 2011). Our work follows up on this research but provides a sys- tematic investigation of how data expansion, lemma- tisation and synonym handling impacts the perfor- mance of a supervised QA engine. 3 Experimental Setup We run our experiments on a dialog engine de- veloped for a serious game called Mission Plastech- nologie. In this game, the player must interact with different virtual humans through a sequence of 12 subdialogs, each of them occurring in a different part of the virtual world. Training Data. The training corpus consists of around 1250 Human-Human dialogues …

Content truncated to 500 words