Improved Semantic Parsers For If-Then Statements

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 726–736, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Improved Semantic Parsers For If-Then Statements I. Beltagy The University of Texas at Austin beltagy@cs.utexas.edu Chris Quirk Microsoft Research chrisq@microsoft.com Abstract Digital personal assistants are becoming both more common and more useful. The major NLP challenge for personal assis- tants is machine understanding: translat- ing natural language user commands into an executable representation. This paper focuses on understanding rules written as If-Then statements, though the techniques should be portable to other semantic pars- ing tasks. We view understanding as struc- ture prediction and show improved mod- els using both conventional techniques and neural network models. We also discuss various ways to improve generalization and reduce overfitting: synthetic training data from paraphrase, grammar combina- tions, feature selection and ensembles of multiple systems. An ensemble of these techniques achieves a new state of the art result with 8% accuracy improvement. 1 Introduction The ability to instruct computers using natural lan- guage clearly allows novice users to better use modern information technology. Work in se- mantic parsing has explored mapping natural lan- guage to some formal domain-specific program- ming languages such as database queries (Woods, 1977; Zelle and Mooney, 1996; Berant et al., 2013; Andreas et al., 2016; Yin et al., 2016), commands to robots (Kate et al., 2005), operat- ing systems (Branavan et al., 2009), and spread- sheets (Gulwani and Marron, 2014). This pa- per explores the use of neural network models (NN) and conventional models for semantic pars- ing. Recently approaches using neural networks have shown great improvements in a number of areas such as parsing (Vinyals et al., 2015), ma- chine translation (Devlin et al., 2014), and image captioning (Karpathy and Fei-Fei, 2015). We are among the first to apply neural network methods to semantic parsing tasks (Grefenstette et al., 2014; Dong and Lapata, 2016). There are several benchmark datasets for se- mantic parsing, the most well known of which is Geoquery (Zelle and Mooney, 1996). We target an If-Then dataset (Quirk et al., 2015) for sev- eral reasons. First, it is both directly applica- ble to the end-user task of training personal dig- ital assistants. Second, the training data, drawn from the site http://ifttt.com, is com- paratively quite large, containing nearly 100,000 recipe-description pairs. That said, it is several or- ders of magnitude smaller than the data for other tasks where neural networks have been successful. Machine translation datasets, for instance, may contain billions of tokens. NN methods appear “data-hungry”. They require larger datasets to out- perform sparse linear approaches with careful fea- ture engineering, as evidenced in work on syntac- tic parsing (Vinyals et al., 2015). This makes it interesting to compare NN models with conven- tional models on this dataset. As in most prior semantic parsing attempts, we model natural language understanding as a struc- ture prediction problem. Each modeling decision predicts some small component of the target struc- ture, conditioned on the …

Content truncated to 500 words