ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2804–2818 August 1–6, 2021. ©2021 Association for Computational Linguistics 2804 ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation Kaushal Kumar Maurya Indian Institute of Technology Hyderabad Hyderabad, India cs18resch11003@iith.ac.in Yoshinobu Kano Shizuoka University, Japan kano@inf.shizuoka.ac.jp Maunendra Sankar Desarkar Indian Institute of Technology Hyderabad Hyderabad, India maunendra@cse.iith.ac.in Kumari Deepshikha NVIDIA, India deepkshikha@gmail.com Abstract Despite the recent advancement in NLP re- search, cross-lingual transfer for natural lan- guage generation is relatively understudied. In this work, we transfer supervision from high resource language (HRL) to multiple low- resource languages (LRLs) for natural lan- guage generation (NLG). We consider four NLG tasks (text summarization, question gen- eration, news headline generation, and dis- tractor generation) and three syntactically di- verse languages, i.e., English, Hindi, and Japanese. We propose an unsupervised cross- lingual language generation framework (called ZmBART) that does not use any parallel or pseudo-parallel/back-translated data. In this framework, we further pre-train mBART sequence-to-sequence denoising auto-encoder model with an auxiliary task using monolin- gual data of three languages. The objective function of the auxiliary task is close to the target tasks which enriches the multi-lingual latent representation of mBART and provides good initialization for target tasks. Then, this model is fine-tuned with task-specific super- vised English data and directly evaluated with low-resource languages in the Zero-shot set- ting. To overcome catastrophic forgetting and spurious correlation issues, we applied freezing model component and data argumen- tation approaches respectively. This simple modeling approach gave us promising results. We experimented with few-shot training (with 1000 supervised data-points) which boosted the model performance further. We performed several ablations and cross-lingual transferabil- ity analysis to demonstrate the robustness of ZmBART. 1 Introduction Recent advancement in natural language generation (NLG) is heavily oriented towards large annotated training data. Such large task-specific annotated News Passage: दि�ण क�ीर के पुलवामा िजले म� सुर�ा बलों के साथ जारी मुठभेड़ म� शु�वार को एक आतंकवादी ढेर हो गया.पुिलस के एक �व�ा ने बताया िक इस मुठभेड़ म� एक आतंकवादी मारा गया है. यह मुठभेड़ अभी जारी है.�व�ा ने बताया िक पुलवामा के च�गाम म� आज सुबह सुर�ा बलों और िछपे �ए आतंकवािदयों के बीच मुठभेड़ शु� हो गई।माना जा रहा है िक गांव म� ल�र-ए-तैयबा के दो आतंकवादी िछपे �ए ह�। (Translation: A militant was killed on Friday in an ongoing encounter with security forces in Pulwama district of eroded Kashmir. A police spokesman said a militant was killed in the encounter. The encounter is still going on, the spokesperson said, adding that an encounter between security forces and hidden militants started this morning at Chandgam in Pulwama. Two LeT militants are believed to be hiding in the village.) Headline (ground truth): क�ीर के पुलवामा म� मुठभेड़, एक आतंकी ढेर (Translation: Encounter in Pulwama, Kashmir, a terrorist killed) Headline (zero-shot generated output:) पुलवामा म� जारी मुठभेड़ म� एक आतंकवादी ढेर (Translation: A terrorist killed in ongoing encounter in Pulwama ) Figure 1: Zero-shot news headline generation from Zm- BART in Hindi language data is …

Content truncated to 500 words