Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1174–1189 August 1–6, 2021. ©2021 Association for Computational Linguistics 1174 Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing Qian Liu†∗, Dejian Yang§, Jiahui Zhang†∗, Jiaqi Guo♦∗, Bin Zhou†, Jian-Guang Lou§ †Beihang University, Beijing, China §Microsoft Research, Beijing, China ♦Xi’an Jiaotong University, Xi’an, China †{qian.liu, 17231043, zhoubin}@buaa.edu.cn; ♦jasperguo2013@stu.xjtu.edu.cn §{dejian.yang, jlou}@microsoft.com Abstract Recent years pretrained language models (PLMs) hit a success on several downstream tasks, showing their power on modeling lan- guage. To better understand and leverage what PLMs have learned, several techniques have emerged to explore syntactic structures entailed by PLMs. However, few efforts have been made to explore grounding capabilities of PLMs, which are also essential. In this paper, we highlight the ability of PLMs to discover which token should be grounded to which con- cept, if combined with our proposed erasing- then-awakening approach. Empirical studies on four datasets demonstrate that our approach can awaken latent grounding which is under- standable to human experts, even if it is not exposed to such labels during training. More importantly, our approach shows great poten- tial to benefit downstream semantic parsing models. Taking text-to-SQL as a case study, we successfully couple our approach with two off-the-shelf parsers, obtaining an absolute im- provement of up to 9.8%. 1 Introduction Recent breakthroughs of Pretrained Language Models (PLMs) such as BERT (Devlin et al., 2019) and GPT3 (Brown et al., 2020) have demonstrated the effectiveness of self-supervised learning for a range of downstream tasks. Without being guided by structural information in training, PLMs show the potential for learning implicit syntactic struc- tures and language semantic, which can be trans- ferred to other tasks. To better understand and leverage what PLMs have learned, several work has emerged to probe or induce syntactic structures from PLMs. According to prior studies (Rogers et al., 2020), most existing work focuses on syn- tactic structures such as part of speech (Liu et al., ∗Work done during an internship at Microsoft Research. The first three authors contributed equally. Reign Monarchs 1789-1802 George 1802-1826 John I 1826-1845 Andrew George Washington what war was george washington associated with? U.S. President Colonial Beach (a). Structured Table (b). Knowledge Base Figure 1: Typical scenarios for grounding, here the lin- guistic tokens “george washington” can be grounded into different real-world concepts. 2019), constituency tree (Wu et al., 2020) and de- pendency tree (Hewitt and Manning, 2019; Jawahar et al., 2019), paying much less attention on lan- guage semantics (Tenney et al., 2019). However, as well known, semantic information is essential for high-level tasks like machine reading compre- hension (Wang and Jiang, 2019). Regarding to language semantics, an impor- tant branch is grounding, which is overlooked by most previous work. Broadly speaking, grounding means “connecting linguistic symbols to real-world perception or actions” (Roy, 2005). It is generally thought to be important for a variety of tasks, such as video descriptions (Zhou et al., 2019), visual question answering (Zhu et al., 2016) and semantic parsing (Guo et …

Content truncated to 500 words