Mixed-Lingual Pre-training for Cross-lingual Summarization

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 536–541 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 536 Mixed-Lingual Pre-training for Cross-lingual Summarization Ruochen Xu∗, Chenguang Zhu∗, Yu Shi, Michael Zeng, Xuedong Huang Microsoft Cognitive Services Research Group {ruox,chezhu,yushi,nzeng,xdh}@microsoft.com Abstract Cross-lingual Summarization (CLS) aims at producing a summary in the target lan- guage for an article in the source lan- guage. Traditional solutions employ a two- step approach, i.e. summarize→translate.translate→summarize or Recently, end-to-end models have achieved better results, but these approaches are mostly limited by their depen- dence on large-scale labeled data. We pro- pose a solution based on mixed-lingual pre- training that leverages both cross-lingual tasks such as translation and monolingual tasks like masked language models. Thus, our model can leverage the massive monolingual data to enhance its modeling of language. Moreover, the architecture has no task-specific compo- nents, which saves memory and increases opti- mization efficiency. We show in experiments that this pre-training scheme can effectively boost the performance of cross-lingual summa- rization. In Neural Cross-Lingual Summariza- tion (NCLS) (Zhu et al., 2019b) dataset, our model achieves an improvement of 2.82 (En- glish to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results. 1 Introduction Text summarization can facilitate the propagation of information by providing an abridged version for long articles and documents. Meanwhile, the globalization progress has prompted a high demand of information dissemination across language bar- riers. Thus, the cross-lingual summarization (CLS) task emerges to provide accurate gist of articles in a foreign language. Traditionally, most CLS methods follow the two- step pipeline approach: either translate the arti- cle into the target language and then summarize it (Leuski et al., 2003), or summarize the article in the source language and then translate it (Wan ∗Equal contribution et al., 2010). Although this method can leverage off-the-shelf summarization and MT models, it suf- fers from error accumulation from two indepen- dent subtasks. Therefore, several end-to-end ap- proaches have been proposed recently (Zhu et al., 2019b; Ouyang et al., 2019; Duan et al., 2019), which conduct both translation and summarization simultaneously. Easy to optimize as these meth- ods are, they typically require a large amount of cross-lingual summarization data, which may not be available especially for low-resource languages. For instance, NCLS (Zhu et al., 2019b) proposes to co-train on monolingual summarization (MS) and machine translation (MT) tasks, both of which require tremendous labeling efforts. On the other hand, the pre-training strategy has proved to be very effective for language understand- ing (Devlin et al., 2018; Holtzman et al., 2019) and cross-lingual learning (Lample and Conneau, 2019; Chi et al., 2019). One of the advantages of pre-training is that many associated tasks are self- learning by nature, which means no labeled data is required. This greatly increases the amount of train- ing data exposed to the model, thereby enhancing its performance on downstream tasks. Therefore, we leverage large-scale …

Content truncated to 500 words