Multi-Task Learning for Cross-Lingual Abstractive Summarization

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3008–3016 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3008 Multi-Task Learning for Cross-Lingual Abstractive Summarization Sho Takase, Naoaki Okazaki Tokyo Institute of Technology {sho.takase@nlp.c, okazaki@c}.titech.ac.jp Abstract We present a multi-task learning framework for cross-lingual abstractive summarization to augment training data. Recent studies constructed pseudo cross-lingual abstractive summarization data to train their neural encoder-decoders. Meanwhile, we introduce existing genuine data such as translation pairs and monolingual abstractive summarization data into training. Our proposed method, Transum, attaches a special token to the beginning of the input sentence to indicate the target task. The special token enables us to incorporate the genuine data into the training data easily. The experimental results show that Transum achieves better performance than the model trained with only pseudo cross-lingual summarization data. In addition, we achieve the top ROUGE score on Chinese-English and Arabic-English abstractive summarization. Moreover, Transum also has a positive effect on machine translation. Experimental results indicate that Transum improves the performance from the strong baseline, Transformer, in Chinese-English, Arabic-English, and English-Japanese translation datasets. Keywords: summarization, machine translation, cross-lingual summarization 1. Introduction Cross-lingual abstractive summarization is the task to generate a summary of a given document in a different target language (Leuski et al., 2003; Wan et al., 2010; Duan et al., 2019; Zhu et al., 2019). This task provides the overview of an article in a foreign language and thus helps readers understand a text written in an unfamiliar language quickly. Early work on cross-lingual abstractive summarization adopted the pipeline approach: either translation of the given document into the target language followed by summarization of the translated document (Leuski et al., 2003) or summarization of the given document fol- lowed by translation of the summary into the target lan- guage (Or˘asan and Chiorean, 2008; Wan et al., 2010). On the other hand, recent studies have applied a neu- ral encoder-decoder model, which is widely used for natural language generation tasks including machine translation (Sutskever et al., 2014) and monolingual ab- stractive summarization (Rush et al., 2015), to generate a summary in the target language from the given docu- ment directly (Duan et al., 2019; Zhu et al., 2019). Such direct generation approaches prevent the error propaga- tion in pipeline methods. Training neural encoder-decoder models requires nu- merous sentence pairs. In fact, Rush et al. (2015) provided 3.8M sentence-summary pairs to train their neural encoder-decoder model for English abstractive summarization, and the following studies used the same training data (Zhou et al., 2017; Kiyono et al., 2017; Cao et al., 2018). However, constructing a large-scale cross- lingual abstractive summarization dataset is much more difficult than collecting monolingual summarization datasets because we require sentence-summary pairs in different languages. To address this issue, recent stud- ies applied a machine translation model to monolingual sentence-summary pairs (Ayana et al., 2018; Duan et al., 2019; Zhu et al., 2019). They used the constructed pseudo dataset to train their neural encoder-decoder models. …

Content truncated to 500 words