Contrastive Aligned Joint Learning for Multilingual Summarization

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2739–2750 August 1–6, 2021. ©2021 Association for Computational Linguistics 2739 Contrastive Aligned Joint Learning for Multilingual Summarization Danqing Wang, Jiaze Chen, Hao Zhou, Xipeng Qiu †, Lei Li ByteDance AI Lab {wangdanqing.122,chenjiaze,zhouhao.nlp,lileilab}@bytedance.com † Fudan University xpqiu@fudan.edu.cn Abstract Multilingual text summarization requires the ability to understand documents in multiple languages and generate summaries in the corresponding language, which poses more challenges on current summarization systems. However, this problem has been rarely stud- ied due to the lack of large-scale supervised summarization data in multiple languages. In this paper, we first provide a large-scale multilingual summarization corpus MLGSum consisting of 1.1 million articles and sum- maries in 12 different languages. Based on it, we develop a unified summarization model to understand the document and gener- ate summaries in different languages. We use the contrastive learning strategy to train our multilingual summarization system (CALMS), which consists of two training objectives, con- trastive sentence ranking (CSR) and sentence aligned substitution (SAS). The two training objectives are designed to share salient infor- mation extractive ability and align sentence- level representation across different languages. Experimental results indicate that CALMS achieves significant improvement over mono- lingual models in all languages. We fur- ther transfer CALMS to other languages and find that it will also benefit similar lan- guages. Our code and dataset are available at https://github.com/brxx122/CALMS. 1 Introduction Automatic text summarization aims at providing a brief summary for a long document. It requires the ability to understand document-level input, catch the main idea of it, and generate a fluent text. Re- cently, monolingual summarization has witnessed great success with the development of new neu- ral systems (Zhong et al., 2020; Wang et al., 2020) and the availability of monolingual pre-training lan- guage models (Kenton and Toutanova, 2019; Liu and Lapata, 2019; Liu et al., 2019; Lewis et al., 2020b). , Inspired by the success of monolingual pre-trained models, researchers further pre-train these models with multiple languages to get the multilingual versions (Huang et al., 2019; Liu et al., 2020; Lewis et al., 2020a), which provide the abil- ities of understanding and generation in different languages. The multilingual pre-training model can be used as the initialization and finetuned for downstream summarization tasks. However, the pre-training phase for language models usually focuses on predicting masked to- kens or denoising the noisy input, both of which are token-level tasks. It lacks the ability to align sentence-level information among languages and to distinguish which information is the most critical for the document-level input. Most previous mul- tilingual summarization models focus on training one model for different language or partly share en- coder/decoder layers (Wang et al., 2018; Lin et al., 2018; Scialom et al., 2020). Cao et al. (2020) and Lewis et al. (2020a) try to train one model for all languages, but they find that although low-resource languages can benefit from the larger training data, the performance of rich-resource languages has been sacrificed. Thus, we want to investigate the following …

Content truncated to 500 words