Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9279 - 9300 December 7-11, 2022 ©2022 Association for Computational Linguistics Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation Tu Vu1,2⋆, Aditya Barua1, Brian Lester1, Daniel Cer1, Mohit Iyyer2, Noah Constant1 Google Research1 University of Massachusetts Amherst2 {ttvu, adityabarua, brianlester, cer, nconstant}@google.com {tuvu, miyyer}@cs.umass.edu Abstract In this paper, we explore the challenging prob- lem of performing a generative task in a tar- get language when labeled data is only avail- able in English, using summarization as a case study. We assume a strict setting with no ac- cess to parallel data or machine translation and find that common transfer learning ap- proaches struggle in this setting, as a genera- tive multilingual model fine-tuned purely on English catastrophically forgets how to gen- erate non-English. Given the recent rise of parameter-efficient adaptation techniques, we conduct the first investigation into how one such method, prompt tuning (Lester et al., 2021), can overcome catastrophic forgetting to enable zero-shot cross-lingual generation. Our experiments show that parameter-efficient prompt tuning provides gains over standard fine-tuning when transferring between less- related languages, e.g., from English to Thai. However, a significant gap still remains be- tween these methods and fully-supervised baselines. To improve cross-lingual transfer further, we explore several approaches, in- cluding: (1) mixing in unlabeled multilingual data, and (2) explicitly factoring prompts into recombinable language and task components. Our approaches can provide further quality gains, suggesting that robust zero-shot cross- lingual generation is within reach. 1 Introduction Cross-lingual language understanding is an impor- tant area of ongoing research (Conneau et al., 2020; Hu et al., 2020; Ruder et al., 2021). With vastly differing amounts of data (both labeled and unla- beled) available across languages, there is signifi- cant value to developing techniques that can trans- fer knowledge from higher-resource languages to improve performance in lower-resource languages. Zero-shot cross-lingual benchmarks push on the ⋆ Work done as a student researcher at Google Brain. Inference time: Apply the resulting LM to summarize articles written in non-English languages (zero-shot cross-lingual) Training time: Adapt a pretrained multilingual LM to English summarization using prompt tuning or model tuning English article: Mask the noise in your ears by turning on background music or other sounds You can use tapes or CDs with “white noise” of the ocean, … English summary: Use calming background sound to drown out the noise. Listen to soothing sounds as you fall asleep … Thai article: กลบเสียงดังในหูโดยเปดเพลง บรรเลงหรือเสียงบรรยากาศคลอไป จะเปดคลิ ปหรือแผน CD ที่เปน … Thai summary: ใชเสียง บรรยากาศชวนสงบใจ. ฟงเสียงขับ กลอมจนหลับไป. Multilingual Language Model (mT5) Figure 1: A demonstration of WIKILINGUA-0, a chal- lenging zero-shot cross-lingual generation (XGEN) task, which requires a model to learn a generative task from labeled data in one language (i.e., English), and then perform the equivalent task in another language at in- ference time. limiting case where no labeled data is available in the target language. Remarkable progress has been made on zero-shot cross-lingual tasks by scal- ing up the size of pre-trained multilingual models …

Content truncated to 500 words