On Evaluating Multilingual Compositional Generalization with Translated Datasets

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1669–1687 July 9-14, 2023 ©2023 Association for Computational Linguistics On Evaluating Multilingual Compositional Generalization with Translated Datasets Zi Wang1,2 and Daniel Hershcovich1 1Department of Computer Science, 2Department of Nordic Studies and Linguistics University of Copenhagen {ziwa, dh}@di.ku.dk Abstract Compositional generalization allows efficient learning and human-like inductive biases. Since most research investigating composi- tional generalization in NLP is done on En- glish, important questions remain underex- plored. Do the necessary compositional gen- eralization abilities differ across languages? Can models compositionally generalize cross- lingually? As a first step to answering these questions, recent work used neural machine translation to translate datasets for evaluating compositional generalization in semantic pars- ing. However, we show that this entails crit- ical semantic distortion. To address this lim- itation, we craft a faithful rule-based transla- tion of the MCWQ dataset (Cui et al., 2022) from English to Chinese and Japanese. Even with the resulting robust benchmark, which we call MCWQ-R, we show that the distribution of compositions still suffers due to linguistic diver- gences, and that multilingual models still strug- gle with cross-lingual compositional general- ization. Our dataset and methodology will be useful resources for the study of cross-lingual compositional generalization in other tasks.1 1 Introduction A vital ability desired for language models is com- positional generalization (CG), the ability to gen- eralize to novel combinations of familiar units (Oren et al., 2020). Semantic parsing enables executable representation of natural language ut- terances for knowledge base question answering (KBQA; Lan et al., 2021). A growing amount of research has been investigating the CG ability of semantic parsers based on carefully constructed datasets, typically synthetic corpora (e.g., CFQ; Keysers et al., 2019) generated based on curated rules, mostly within monolingual English scenar- ios. As demonstrated by Perevalov et al. (2022), 1The dataset, trained models and code for the experiments and dataset generation are available at https://github.com/ ziwang-klvk/CFQ-RBMT. NEURAL-BASED TRANSLATION: SOURCE: Did Erika Mann’s spouse executive produce Friedemann Bach TARGET: 艾莉卡·曼的配偶执行官 制作 了 弗里德曼·巴赫 吗 RULE-BASED TRANSLATION: SOURCE: Did Erika Mann’s spouse executive produce Friedemann Bach TARGET: 艾莉卡·曼的配偶 NP1 执行制作 V 了 弗里德曼·巴赫 NP2 吗 SPARQL QUERY: ASK WHERE { wd:Q829979 wdt:P1431 ?x0 . ?x0 wdt:P26 wd:Q61597 . FILTER ( ?x0 != wd:Q61597 )} Figure 1: Example of neural machine translation (NMT, from MCWQ, top) and rule-based translation (from MCWQ-R, middle) from English to Chinese. The com- positions correctly captured by the translation system and the correspondences in the SPARQL query (bottom) are highlighted in the same color, while errors are in red. NMT often diverges semantically from the query: here, the compound “executive produce” is split. RBMT per- forms well due to awareness of grammar constituents. resource scarcity for many languages largely pre- clude their speakers’ access to knowledge bases (even for languages they include), and KBQA in multilingual scenarios is barely researched mainly due to lack of corresponding benchmarks. Cui et al. (2022) proposed Multilingual Com- positional Wikidata Questions (MCWQ) …

Content truncated to 500 words