Compositional Generalization in Multilingual Semantic Parsing over Wikidata

Compositional Generalization in Multilingual Semantic Parsing over Wikidata Ruixiang Cui, Rahul Aralikatte, Heather Lent, and Daniel Hershcovich Department of Computer Science University of Copenhagen, Denmark {rc, rahul, hcl, dh}@di.ku.dk Abstract Semantic parsing (SP) allows humans to lever- age vast knowledge resources through natural interaction. However, parsers are mostly de- signed for and evaluated on English resources, such as CFQ (Keysers et al., 2020), the cur- rent standard benchmark based on English data generated from grammar rules and ori- ented towards Freebase, an outdated knowl- edge base. We propose a method for creating a multilingual, parallel dataset of question-query pairs, grounded in Wikidata. We introduce such a dataset, which we call Multilingual Compositional Wikidata Questions (MCWQ), and use it to analyze the compositional gen- eralization of semantic parsers in Hebrew, Kannada, Chinese, and English. While within- language generalization is comparable across languages, experiments on zero-shot cross- lingual transfer demonstrate that cross-lingual compositional generalization fails, even with state-of-the-art pretrained multilingual encod- ers. Furthermore, our methodology, dataset, and results will facilitate future research on SP in more realistic and diverse settings than has been possible with existing resources. 1 Introduction Semantic parsers grounded in knowledge bases (KBs) enable knowledge base question answer- ing (KBQA) for complex questions. Many seman- tic parsers are grounded in KBs such as Freebase (Bollacker et al., 2008), DBpedia (Lehmann et al., 2015), and Wikidata (Pellissier Tanon et al., 2016), and models can learn to answer questions about unseen entities and properties (Herzig and Berant, 2017; Cheng and Lapata, 2018, Shen et al., 2019; Sas et al., 2020). An important desired abil- ity is compositional generalization—the ability to generalize to unseen combinations of known components (Oren et al., 2020; Kim and Linzen, 2020). One of the most widely used datasets for mea- suring compositional generalization in KBQA is CFQ (Compositional Freebase Questions; Keysers et al., 2020), which was generated using grammar rules, and is based on Freebase, an outdated and unmaintained English-only KB. While the need to expand language technology to many languages is widely acknowledged (Joshi et al., 2020), the lack of a benchmark for compositional generaliza- tion in multilingual semantic parsing (SP) hinders KBQA in languages other than English. Further- more, progress in both SP and KB necessitates that benchmarks can be reused and adapted for future methods. Wikidata is a multilingual KB, with entity and property labels in a multitude of languages. It has grown continuously over the years and is an important complement to Wikipedia. Much effort has been made to migrate Freebase data to Wiki- data (Pellissier Tanon et al., 2016; Diefenbach et al., 2017; Hogan et al., 2021), but only in En- glish. Investigating compositional generalization in cross-lingual SP requires a multilingual data- set, a gap we address in this work. We leverage Wikidata and CFQ to create Multilingual Compositional Wikidata Questions (MCWQ), a new multilingual dataset of compo- sitional questions grounded in Wikidata (see Fig- ure 1 for an example). Beyond the original English, an Indo-European language using the Latin script, we …

Content truncated to 500 words