XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15918–15947 July 9-14, 2023 ©2023 Association for Computational Linguistics XSEMPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations Yusen Zhang1 Jun Wang2 Zhiguo Wang2 Rui Zhang1 1Penn State University 2AWS AI Labs {yfz5488,rmz5227}@psu.edu, {juwanga,zhiguow}@amazon.com Abstract Cross-Lingual Semantic Parsing (CLSP) aims to translate queries in multiple natural lan- guages (NLs) into meaning representations (MRs) such as SQL, lambda calculus, and logic forms. However, existing CLSP models are separately proposed and evaluated on datasets of limited tasks and applications, impeding a comprehensive and unified evaluation of CLSP on a diverse range of NLs and MRs. To this end, we present XSEMPLR, a unified bench- mark for cross-lingual semantic parsing fea- tured with 22 natural languages and 8 mean- ing representations by examining and select- ing 9 existing datasets to cover 5 tasks and 164 domains. We use XSEMPLR to conduct a comprehensive benchmark study on a wide range of multilingual language models includ- ing encoder-based models (mBERT, XLM-R), encoder-decoder models (mBART, mT5), and decoder-based models (Codex, BLOOM). We design 6 experiment settings covering various lingual combinations (monolingual, multilin- gual, cross-lingual) and numbers of learning samples (full dataset, few-shot, and zero-shot). Our experiments show that encoder-decoder models (mT5) achieve the highest performance compared with other popular models, and mul- tilingual training can further improve the aver- age performance. Notably, multilingual large language models (e.g., BLOOM) are still inad- equate to perform CLSP tasks. We also find that the performance gap between monolin- gual training and cross-lingual transfer learn- ing is still significant for multilingual models, though it can be mitigated by cross-lingual few- shot training. Our dataset and code are avail- able at https://github.com/psunlpgroup/ XSemPLR. 1 Introduction Cross-Lingual Semantic Parsing (CLSP) aims to translate queries in multiple natural languages (NLs) into meaning representations (MRs) (Li et al., 2020; Xu et al., 2020a; Dou et al., 2022; Sher- borne and Lapata, 2021, 2022). As demonstrated in Figure 1, Cross-Lingual Semantic Parsing covers natural languages for geographically diverse users and various meaning representations, empowering applications such as natural language interfaces to databases, question answering over knowledge graphs, virtual assistants, smart home device con- trol, human-robot interaction, and code generation. However, current research on CLSP has three drawbacks. First, most existing research focuses on semantic parsing in English (Zelle and Mooney, 1996; Wang et al., 2015; Yu et al., 2018), limiting the development of multilingual information ac- cess systems for users in other languages. Second, current datasets have a poor coverage of NLs and MRs. Although there are encouraging efforts in de- veloping CLSP models (Li et al., 2020; Dou et al., 2022; Sherborne and Lapata, 2022), their experi- ments only cover a few NLs and MRs, impeding comprehensive and unified evaluation on a diverse range of tasks. Third, due to the lack of a compre- hensive CLSP benchmark, the performance of mul- tilingual language models on CLSP is understudied. Some pretrained language models are proposed to …

Content truncated to 500 words