Transfer Learning for Sequence Generation: from Single-source to Multi-source

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5738–5750 August 1–6, 2021. ©2021 Association for Computational Linguistics 5738 Transfer Learning for Sequence Generation: from Single-source to Multi-source Xuancheng Huang1, Jingfang Xu4, Maosong Sun1,3, and Yang Liu1,2,3∗ 1Dept. of Comp. Sci. & Tech., BNRist Center, Institute for AI, Tsinghua University 2Institute for AI Industry Research, Tsinghua University, Beijing, China 3Beijing Academy of Artificial Intelligence 4Sogou Inc., Beijing, China Abstract Multi-source sequence generation (MSG) is an important kind of sequence generation tasks that takes multiple sources, including automatic post-editing, multi-source transla- tion, multi-document summarization, etc. As MSG tasks suffer from the data scarcity prob- lem and recent pretrained models have been proven to be effective for low-resource down- stream tasks, transferring pretrained sequence- to-sequence models to MSG tasks is essential. Although directly finetuning pretrained mod- els on MSG tasks and concatenating multiple sources into a single long sequence is regarded as a simple method to transfer pretrained mod- els to MSG tasks, we conjecture that the di- rect finetuning method leads to catastrophic forgetting and solely relying on pretrained self- attention layers to capture cross-source infor- mation is not sufficient. Therefore, we propose a two-stage finetuning method to alleviate the pretrain-finetune discrepancy and introduce a novel MSG model with a fine encoder to learn better representations in MSG tasks. Experi- ments show that our approach achieves new state-of-the-art results on the WMT17 APE task and multi-source translation task using the WMT14 test set. When adapted to document- level translation, our framework outperforms strong baselines significantly.1 1 Introduction Thanks to the continuous representations widely used across text, speech, and image, neural net- works that accept multiple sources as input have gained increasing attention in the community (Ive et al., 2019; Dupont and Luettin, 2000). For ex- ample, multi-modal inputs that are complementary have proven to be helpful for many sequence gener- ation tasks such as question answering (Antol et al., ∗Corresponding author: Yang Liu 1The source code is available at https://github. com/THUNLP-MT/TRICE Pretraining Single-source SG Multi-source SG AutoEncoding BERT-FUSED DUALBERT (e.g., BERT) (Zhu et al., 2019) (Correia and Martins, 2019) Seq2Seq MBART-TRANS this work (e.g., BART) (Liu et al., 2020) Table 1: Comparison of various approaches to trans- ferring pretrained models to single-source and multi- source sequence generation tasks. Different from prior studies, this work aims at transferring pre- trained sequence-to-sequence models to multi-source sequence generation tasks. 2015), machine translation (Huang et al., 2016), and speech recognition (Dupont and Luettin, 2000). In natural language processing, multiple textual inputs have also been shown to be valuable for se- quence generation tasks such as multi-source trans- lation (Zoph and Knight, 2016), automatic post- editing (Chatterjee et al., 2017), multi-document summarization (Haghighi and Vanderwende, 2009), system combination for NMT (Huang et al., 2020), and document-level machine translation (Wang et al., 2017). We refer to this kind of tasks as multi-source sequence generation (MSG). Unfortunately, MSG tasks face a severe chal- lenge: …

Content truncated to 500 words