Exploring the Secrets Behind the Learning Difficulty of Meaning Representations for Semantic Parsing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3616 - 3625 December 7-11, 2022 ©2022 Association for Computational Linguistics Exploring the Secrets Behind the Learning Difficulty of Meaning Representations for Semantic Parsing Zhenwen Li♢, Jiaqi Guo§ ∗, Qian Liu† ∗, Jian-Guang Lou♡, Tao Xie♢ ∗ ♢Peking University, Beijing, China; §Xi’an Jiaotong University, Xi’an, China †Sea AI Lab, Singapore; ♡Microsoft Research Asia, Beijing, China lizhenwen@pku.edu.cn, jasperguo2013@stu.xjtu.edu.cn, liuqian@sea.com jlou@microsoft.com, taoxie@pku.edu.cn Abstract Previous research has shown that the design of Meaning Representation (MR) greatly influ- ences model performance of a neural semantic parser. Therefore, designing a good MR is a long-term goal for semantic parsing. However, it is still an art as there is no quantitative indi- cator that can tell us which MR among a set of candidates may have the best final model per- formance. In practice, in order to select an MR, researchers often have to go through the whole training-testing process for all MR candidates, and the process often costs a lot. In this paper, we propose a data-aware metric called ISS (de- noting incremental structural stability) of MRs, and demonstrate that ISS is highly correlated with model performance. The finding shows that ISS can be used as an indicator for design- ing MRs to avoid the costly training-testing process. 1 Introduction Semantic parsing is the task of transforming a natu- ral language utterance into a Meaning Representa- tion (MR), whose accuracy has been significantly improved recently (Dong and Lapata, 2016; Jia and Liang, 2016; Yin and Neubig, 2017; Dong and La- pata, 2018; Iyer et al., 2017). For example, in a semantic parsing dataset of Text-to-SQL, SQL is the meaning representation. Previous work (Guo et al., 2019; Herzig et al., 2021; Kate et al., 2005) has shown that the de- sign of MR significantly influences model perfor- mance of a neural semantic parser. As shown in Table 1, even with the same neural network archi- tecture, using SQL as the MR gets an execution accuracy of 61.0 on the ATIS dataset and 72.1 on the GeoQuery dataset, while using Lambda as the MR reaches 82.1 and 80.7, respectively. The dif- ference of model performance between the MRs ∗ The corresponding author is Tao Xie. Jiaqi Guo and Qian Liu contributed to this work during their internships at Microsoft Research Asia. Table 1: The execution accuracy of a neural seq2prod model for different MRs on three semantic parsing datasets: ATIS, GeoQuery, and Job. The results come from the work of Guo et al. (2020). Dataset SQL Lambda Prolog FunQL ATIS 61.0 82.1 79.2 82.7 GeoQuery 72.1 80.7 79.5 86.2 Job 87.6 91.0 84.4 92.4 even reaches 20% under the same conditions.1 The difference in model performance between MRs in- dicates that different MRs bring different levels of learning difficulties to a neural model. In order to reduce learning difficulty of MRs, some researchers have put efforts to design cus- tomized MRs for their tasks. For example, Guo et al. (2019) propose a new MR …

Content truncated to 500 words