Scheduled Multi-task Learning for Neural Chat Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4375 - 4388 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Scheduled Multi-task Learning for Neural Chat Translation Yunlong Liang1∗, Fandong Meng2, Jinan Xu1†, Yufeng Chen1 and Jie Zhou2 1Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China {yunlongliang,jaxu,chenyf}@bjtu.edu.cn {fandongmeng,withtomzhou}@tencent.com Abstract Neural Chat Translation (NCT) aims to trans- late conversational text into different languages. Existing methods mainly focus on modeling the bilingual dialogue characteristics (e.g., coher- ence) to improve chat translation via multi-task learning on small-scale chat translation data. Although the NCT models have achieved im- pressive success, it is still far from satisfac- tory due to insufficient chat translation data and simple joint training manners. To ad- dress the above issues, we propose a scheduled multi-task learning framework for NCT. Specif- ically, we devise a three-stage training frame- work to incorporate the large-scale in-domain chat translation data into training by adding a second pre-training stage between the orig- inal pre-training and fine-tuning stages. Fur- ther, we investigate where and how to schedule the dialogue-related auxiliary tasks in multiple training stages to effectively enhance the main chat translation task. Extensive experiments on four language directions (English↔Chinese and English↔German) verify the effectiveness and superiority of the proposed approach. Ad- ditionally, we will make the large-scale in- domain paired bilingual dialogue dataset pub- licly available for the research community.1 1 Introduction A cross-lingual conversation involves speakers in different languages (e.g., one speaking in Chinese and another in English), where a chat translator can be applied to help them communicate in their native languages. The chat translator bilaterally converts the language of bilingual conversational text, e.g. from Chinese to English and vice versa (Wang et al., 2016a; Farajian et al., 2020; Liang et al., 2021a, 2022). ∗Work was done when Yunlong was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. †Jinan Xu is the corresponding author. 1The code and in-domain data are publicly available at: https://github.com/XL2248/SML Figure 1: The overall three-stage training framework. Generally, since the bilingual dialogue corpus is scarce, researchers (Bao et al., 2020; Wang et al., 2020; Liang et al., 2021a,d) resort to making use of the large-scale general-domain data through the pre- training-then-fine-tuning paradigm as done in many context-aware neural machine translation models (Tiedemann and Scherrer, 2017; Maruf and Haf- fari, 2018; Miculicich et al., 2018; Tu et al., 2018; Voita et al., 2018, 2019a,b; Yang et al., 2019; Wang et al., 2019; Maruf et al., 2019; Ma et al., 2020, etc), having made significant progress. However, conventional pre-training on large-scale general- domain data usually learns general language pat- terns, which is also aimless for capturing the useful dialogue context to chat translation, and fine-tuning usually suffers from insufficient supervised data (about 10k bilingual dialogues). Some studies (Gu et al., 2020; Gururangan et al., 2020; Liu et al., 2021; Moghe et al., 2020; Wang et al., 2020; …

Content truncated to 500 words