Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 991 - 1000 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension Linjuan Wu1∗, Shaojuan Wu1∗, Xiaowang Zhang1,2, Deyi Xiong1†, Shizhan Chen1†, Zhiqiang Zhuang1, Zhiyong Feng1 1College of Intelligence and Computing, Tianjin University, Tianjin, China, 300350 2Tianjin University-Aishu Data Intelligence Joint Laboratory, Tianjin, China Tianjin_Research@aishu.cn {wulinjuan1997,shaojuanwu,dyxiong,shizhan}@tju.edu.cn Abstract Multilingual pre-trained models are able to zero-shot transfer knowledge from rich- resource to low-resource languages in machine reading comprehension (MRC). However, in- herent linguistic discrepancies in different lan- guages could make answer spans predicted by zero-shot transfer violate syntactic constraints of the target language. In this paper, we pro- pose a novel multilingual MRC framework equipped with a Siamese Semantic Disentan- glement Model (S2DM) to disassociate seman- tics from syntax in representations learned by multilingual pre-trained models. To explicitly transfer only semantic knowledge to the tar- get language, we propose two groups of losses tailored for semantic and syntactic encoding and disentanglement. Experimental results on three multilingual MRC datasets (i.e., XQuAD, MLQA, and TyDi QA) demonstrate the effec- tiveness of our proposed approach over models based on mBERT and XLM-100. 1 Introduction Multilingual pre-trained language models (PLMs) (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020) have been widely explored in cross-lingual understanding tasks. However, zero-shot transfer method based on multilingual PLMs does not work well for low-resource lan- guage MRC. Such multilingual MRC models could roughly detect answer spans but may fail to pre- dict the precise boundaries of answers (Yuan et al., 2020). In order to address this issue, existing methods mainly resort to external resources. Based on the finding that 70% of answer spans are language- specific phrases (e.g., named entities, noun phrases) in MLQA (Lewis et al., 2020), Yuan et al. (2020) propose an additional language-specific knowledge *These authors contributed equally to this work and should be considered co-first authors. †Corresponding authors. [Passage] [XQuAD] A growing cause of concern are that attacks on teachers in Welsh schools which reached an all-time high between 2005 and 2010.[...] [Question] When were attacks on teachers the highest? [Answer - ground truth] between 2005 and 2010 [the answer is a prepositional phrase containing a noun phrase] ROOT VP NP PP DT JJ IN NP CD an high between 2005 [Passage] [BiPaR] 韦小宝见蛾眉刺上闪出的月光不住晃动，有时直掠到他脸上[...] As Trinket watched, the gleam of reflected light from the stiletto began to jump and waver.[...] [Question] 什么在晃动？ What's shaking? [Answer - ground truth] 月光 the gleam of reflected light [Answer - model prediction] 月光不住 [the answer violating syntactic constituent boundaries] (a) (b) ROOT IP NP VP NN ADVP 月光 AD VP VV IP 住 VP 晃动 不 VV NP CP DEC IP VP 闪出 VV 的 CD 2010 VBD JJ all-time reached CC and Figure 1: Relations between answer spans and syntactic constituents. (a) An example from XQuAD (Artetxe et al., 2020) where the ground-truth answer is a …

Content truncated to 500 words