Benchmarking Meaning Representations in Neural Semantic Parsing

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1520–1540, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1520 Benchmarking Meaning Representations in Neural Semantic Parsing Jiaqi Guo1∗, Qian Liu2∗, Jian-Guang Lou3, Zhenwen Li4 Xueqing Liu5, Tao Xie4, Ting Liu1 1Xi’an Jiaotong University, Xi’an, China 2Beihang University, Beijing, China 3Microsoft Research, Beijing, China 4Peking University, Beijing, China 5Stevens Institute of Technology, Hoboken, New Jersey, USA jasperguo2013@stu.xjtu.edu.cn, qian.liu@buaa.edu.cn jlou@microsoft.com, {lizhenwen, taoxie}@pku.edu.cn xueqing.liu@stevens.edu, tingliu@mail.xjtu.edu.cn Abstract Meaning representation is an important com- ponent of semantic parsing. Although re- searchers have designed a lot of meaning rep- resentations, recent work focuses on only a few of them. Thus, the impact of meaning representation on semantic parsing is less un- derstood. Furthermore, existing work’s per- formance is often not comprehensively evalu- ated due to the lack of readily-available execu- tion engines. Upon identifying these gaps, we propose UNIMER, a new unified benchmark on meaning representations, by integrating ex- isting semantic parsing datasets, completing the missing logical forms, and implementing the missing execution engines. The resulting unified benchmark contains the complete enu- meration of logical forms and execution en- gines over three datasets × four meaning rep- resentations. A thorough experimental study on UNIMER reveals that neural semantic pars- ing approaches exhibit notably different per- formance when they are trained to generate different meaning representations. Also, pro- gram alias and grammar rules heavily impact the performance of different meaning repre- sentations. Our benchmark, execution engines and implementation can be found on: https: //github.com/JasperGuo/Unimer. 1 Introduction A remarkable vision of artificial intelligence is to enable human interactions with machines through natural language. Semantic parsing has emerged as a key technology for achieving this goal. In general, semantic parsing aims to transform a nat- ural language utterance into a logic form, i.e., a formal, machine-interpretable meaning repre- sentation (MR) (Zelle and Mooney, 1996; Dahl et al., 1994).1 Thanks to the recent development ∗Work done during an internship at Microsoft Research. 1In this paper, we focus on grounded semantic parsing, where meaning representations are grounded to specific knowl- MR Geo ATIS Job Prolog - - 91.4 Lambda 90.4 91.3 85.0 FunQL 92.5 - - SQL 78.0 69.0 - Prolog 89.6 - 92.1 Lambda - - - FunQL - - - SQL 82.5 79.2 - Table 1: State-of-the-art performance for MRs on Geo, ATIS, and Job. The top table shows exact-match accu- racy whereas the bottom table shows execution-match accuracy. Most existing work focuses on evaluating only a small subset of dataset × MR pairs, leaving most of the table unexplored. A finer-grained table is avail- able in the supplementary material (Table 8). of neural networks techniques, significant improve- ments have been made in semantic parsing perfor- mance (Jia and Liang, 2016; Yin and Neubig, 2017; Dong and Lapata, 2018; Shaw et al., 2019). Despite the advancement in performance, we identify three important biases in existing work’s evaluation methodology. First, although multiple MRs are proposed, most existing work is evaluated on only one or two of …

Content truncated to 500 words