SemEval-2014 Task 2: Grammar Induction for Spoken Dialogue Systems

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 9–16, Dublin, Ireland, August 23-24, 2014. SemEval-2014 Task 2: Grammar Induction for Spoken Dialogue Systems Ioannis Klasinas1, Elias Iosif2,4, Katerina Louka3, Alexandros Potamianos2,4 1 School of ECE, Technical University of Crete, Chania 73100, Greece 2 School of ECE, National Technical University of Athens, Zografou 15780, Greece 3 Voiceweb S.A., Athens 15124, Greece 4 Athena Research Center, Marousi 15125, Greece iklasinas@isc.tuc.gr,{iosife,potam}@telecom.tuc.gr,klouka@voiceweb.eu Abstract In this paper we present the SemEval- 2014 Task 2 on spoken dialogue gram- mar induction. The task is to classify a lexical fragment to the appropriate se- mantic category (grammar rule) in order to construct a grammar for spoken dia- logue systems. We describe four sub- tasks covering two languages, English and Greek, and three speech application do- mains, travel reservation, tourism and fi- nance. The classification results are com- pared against the groundtruth. Weighted and unweighted precision, recall and f- measure are reported. Three sites partic- ipated in the task with five systems, em- ploying a variety of features and in some cases using external resources for training. The submissions manage to significantly beat the baseline, achieving a f-measure of 0.69 in comparison to 0.56 for the base- line, averaged across all subtasks. 1 Introduction This task aims to foster the application of com- putational models of lexical semantics to the field of spoken dialogue systems (SDS) for the problem of grammar induction. Grammars constitute a vi- tal component of SDS representing the semantics of the domain of interest and allowing the system to correctly respond to a user’s utterance. The task has been developed in tight collabo- ration between the research community and com- mercial SDS grammar developers, under the aus- pices of the EU-IST PortDial project1. Among the This work is licensed under a Creative Commons Attribution 4.0 International Licence. Page numbers and proceedings footer are added by the organisers. Licence details: http: //creativecommons.org/licenses/by/4.0/ 1http://www.portdial.eu/ project aims is to help automate the grammar de- velopment and localization process. Unlike previ- ous approaches (Wang and Acero, 2006; Cramer, 2007) that have focused on full automation, Port- Dial adopts a human-in-the-loop approach were a developer bootstraps each grammar rule or re- quest type with a few examples (use cases) and then machine learning algorithms are used to pro- pose grammar rule enhancements to the developer. The enhancements are post-edited by the devel- oper and new grammar rule suggestions are pro- posed by the system, in an iterative fashion un- til a grammar of sufficient quality is achieved. In this task, we focus on a snapshot of this process, where a portion of the grammar is already induced and post-edited by the developer and new candi- date fragments are rolling in order to be classified to an existing rule (or rejected). The goal is to de- velop machine learning algorithms for classifying candidate lexical fragments to the correct grammar rule (semantic category). The task is equally rel- evant for both finite-state machine and statistical grammar induction. …

Content truncated to 500 words