SemEval-2014 Task 6: Supervised Semantic Parsing of Robotic Spatial Commands

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 45–53, Dublin, Ireland, August 23-24, 2014. SemEval-2014 Task 6: Supervised Semantic Parsing of Robotic Spatial Commands Kais Dukes School of Computing, University of Leeds Leeds LS2 9JT, United Kingdom sckd@leeds.ac.uk Abstract SemEval-2014 Task 6 aims to advance semantic parsing research by providing a high-quality annotated dataset to com- pare and evaluate approaches. The task focuses on contextual parsing of robotic commands, in which the additional con- text of spatial scenes can be used to guide a parser to control a robot arm. Six teams submitted systems using both rule-based and statistical methods. The best per- forming (hybrid) system scored 92.5% and 90.5% for parsing with and without spatial context. However, the best per- forming statistical system scored 87.35% and 60.84% respectively, indicating that generalized understanding of commands given to a robot remains challenging, de- spite the fixed domain used for the task. 1 Introduction Semantic parsers analyze sentences to produce formal meaning representations that are used for the computational understanding of natural lan- guage. Recently, state-of-the-art semantic pars- ing methods have used for a variety of applica- tions, including question answering (Kwiat- kowski et al., 2013; Krishnamurthy and Mitchell, 2012), dialog systems (Artzi and Zettlemoyer, 2011), entity relation extraction (Kate and Mooney, 2010) and robotic control (Tellex, 2011; Kim and Mooney, 2012). Different parsers can be distinguished by the level of supervision they require during training. Fully supervised training typically requires an annotated dataset that maps natural language (NL) to a formal meaning representation such as logical form. However, because annotated data is often not available, a recent trend in semantic parsing research has been to eschew supervised training in favour of either unsupervised or weakly-supervised methods that utilize addi- tional information. For example, Berant and Li- ang (2014) use a dataset of 5,810 question- answer pairs without annotated logical forms to induce a parser for a question-answering system. In comparison, Poon (2013) converts NL ques- tions into formal queries via indirect supervision through database interaction. In contrast to previous work, the shared task described in this paper uses the Robot Com- mands Treebank (Dukes, 2013a), a new dataset made available for supervised semantic parsing. The chosen domain is robotic control, in which NL commands are given to a robot arm used to manipulate shapes on an 8 x 8 game board. De- spite the fixed domain, the task is challenging as correctly parsing commands requires understand- ing spatial context. For example, the command in Figure 1 may have several plausible interpreta- tions, given different board configurations. ‘Move the pyramid on the blue cube on the gray one.’ Figure 1: Example scene with a contextual spatial command from the Robot Commands Treebank. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0 45 The task is inspired by the classic AI system SHRLDU, which responded to NL commands to control a robot for a similar game board (Wino- grad, 1972), although that system …

Content truncated to 500 words