Reranking for Neural Semantic Parsing

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4553–4559 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4553 Reranking for Neural Semantic Parsing Pengcheng Yin Language Technologies Institute Carnegie Mellon University pcyin@cs.cmu.edu Graham Neubig Language Technologies Institute Carnegie Mellon University gneubig@cs.cmu.edu Abstract Semantic parsing considers the task of trans- ducing natural language (NL) utterances into machine executable meaning representations (MRs). While neural network-based seman- tic parsers have achieved impressive improve- ments over previous methods, results are still far from perfect, and cursory manual inspec- tion can easily identify obvious problems such as lack of adequacy or coherence of the gen- erated MRs. This paper presents a simple approach to quickly iterate and improve the performance of an existing neural semantic parser by reranking an n-best list of predicted MRs, using features that are designed to fix observed problems with baseline models. We implement our reranker in a competitive neu- ral semantic parser and test on four semantic parsing (GEO, ATIS) and Python code gen- eration (DJANGO, CONALA) tasks, improv- ing the strong baseline parser by up to 5.7% absolute in BLEU (CONALA) and 2.9% in accuracy (DJANGO), outperforming the best published neural parser results on all four datasets.1 1 Introduction Semantic parsing is the task of mapping a nat- ural language utterance into machine executable meaning representations (e.g., Python code). Re- cent years have witnessed a burgeoning of ap- plying neural network architectures for semantic parsing, from sequence-to-sequence models (Jia and Liang, 2016; Ling et al., 2016; Liang et al., 2017; Suhr et al., 2018), to more complex pars- ing paradigms guided by the structured topolo- gies of target meaning representations (Xiao et al. (2016); Dong and Lapata (2016); Yin and Neubig (2017); Rabinovich et al. (2017); Krishnamurthy et al. (2017); Zhong et al. (2017); Dong and Lap- ata (2018); Iyer et al. (2018), inter alia). 1Code is available at pcyin.me/reranking. download the file from url `url` and save it under file `file_name` json.loads(['url', 'file_name', 'file_name']) urllib.request.urlretrieve('url', 'r') urllib.request.urlretrieve('url', 'file_name') r = urllib.request.urlretrieve(str_0) Input Utterance System Predictions (n-best list of MRs) z1 z2 Reranker Output urllib.request.urlretrieve('url', 'file_name') z9 … z3 z 7! x : −35.2 z $ x : −3.4 z 7! x : −31.4 z $ x : −0.7 z $ x : −0.8 z $ x : −5.8 z 7! x : −49.8 x z3 z 7! x : −34.7 Figure 1: Illustration of the reranker with a real example from the CONALA code generation task (Yin et al., 2018a) with reconstruction (z �→ x) and discriminative matching (x ↔ z) scores. While neural network-based semantic parsers have achieved impressive results, there is still room for improvement. A pilot analysis of incor- rect predictions from a competitive neural seman- tic parser, TRANX (Yin and Neubig, 2018) indi- cates an obvious issue of incoherence. In the real example in Figure 1, top prediction z1 is semanti- cally incoherent with the intent expressed in the ut- terance. Perhaps a more …

Content truncated to 500 words