Improving Zero-Shot Cross-lingual Transfer Between Closely Related Languages by Injecting Character-Level Noise

Findings of the Association for Computational Linguistics: ACL 2022, pages 4074 - 4083 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Improving Zero-Shot Cross-lingual Transfer Between Closely Related Languages by Injecting Character-Level Noise Noëmi Aepli1 and Rico Sennrich1,2 1Department of Computational Linguistics, University of Zurich 2School of Informatics, University of Edinburgh {naepli,sennrich}@cl.uzh.ch Abstract Cross-lingual transfer between a high-resource language and its dialects or closely related lan- guage varieties should be facilitated by their similarity. However, current approaches that operate in the embedding space do not take surface similarity into account. This work presents a simple yet effective strategy to im- prove cross-lingual transfer between closely related varieties. We propose to augment the data of the high-resource source language with character-level noise to make the model more robust towards spelling variations. Our strat- egy shows consistent improvements over sev- eral languages and tasks: Zero-shot transfer of POS tagging and topic identification between language varieties from the Finnic, West and North Germanic, and Western Romance lan- guage branches. Our work provides evidence for the usefulness of simple surface-level noise in improving transfer between language vari- eties. 1 Introduction Recent research has achieved impressive results in zero-shot cross-lingual transfer based on multi- lingual pre-training (Devlin et al., 2019; Conneau and Lample, 2019) or monolingual transfer of em- beddings (Artetxe et al., 2020). However, these methods require large amounts of unlabeled data in the target language (Lauscher et al., 2020) and do not take into account surface similarity between languages except for the sharing of subword units in multilingual models. For the transfer between closely related languages and dialects, we deem it desirable to exploit the similarity of surface rep- resentations. Specifically, we target orthographic variations that commonly result from pronuncia- tion differences between closely related languages. 1 1Note that there are also differences on different levels as described in Hollenstein and Aepli (2014) and partly observ- able in Figure 1 which illustrates a German example sentence with a closely related variant. Figure 1: Swiss German (GSW) sentence with corre- sponding standard German (DE) and English (EN) trans- lations. The sentence shows various spelling differences on the word level, and reordering occurs on the sentence level due to different past-tense formation. In this paper, we propose to augment the training data of a high-resource language with character- level noise to simulate spelling variations and thus facilitate generalization to closely related2 low- resource languages. We test this strategy on two tasks and several language regions. The considered tasks are part-of- speech (POS) tagging on the word level and topic classification on the sentence level; the languages are from the Finnic, West and North Germanic, and Western Romance language branches. We observe that our baseline method for cross-lingual transfer learns undesirable heuristics, e.g., assigning un- seen words to open word classes in POS tagging and that injecting noise reduces this bias. Our ex- periments show absolute accuracy improvements between 1.4 and 22 percentage points over the state of the art, providing evidence that a simple data- …

Content truncated to 500 words