BJTU-WeChat’s Systems for the WMT22 Chat Translation Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 955 - 961 December 7-8, 2022 ©2022 Association for Computational Linguistics BJTU-WeChat’s Systems for the WMT22 Chat Translation Task Yunlong Liang1∗, Fandong Meng2, Jinan Xu1†, Yufeng Chen1 and Jie Zhou2 1Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China {yunlongliang,jaxu,chenyf}@bjtu.edu.cn {fandongmeng,withtomzhou}@tencent.com Abstract This paper introduces the joint submission of the Beijing Jiaotong University and WeChat AI to the WMT’22 chat translation task for English⇔German. Based on the Trans- former (Vaswani et al., 2017), we apply several effective variants. In our experiments, we uti- lize the pre-training-then-fine-tuning paradigm. In the first pre-training stage, we employ data filtering and synthetic data generation (i.e., back-translation, forward-translation, and knowledge distillation). In the second fine- tuning stage, we investigate speaker-aware in- domain data generation, speaker adaptation, prompt-based context modeling, target denois- ing fine-tuning (Meng et al., 2020), and boosted self-COMET-based model ensemble. Our sys- tems achieve 0.810 and 0.946 COMET (Rei et al., 2020) scores1 on English→German and German→English, respectively. The COMET scores of English→German and German→English are the highest among all submissions. 1 Introduction We participate in the WMT 2022 shared task on chat translation in two language directions, English→German and German→English. In this year’s chat translation task, we apply the two-stage training strategy. In the first stage, we investi- gate model architecture and data augmentation. In the second stage, we mainly focus on exploit- ing speaker-aware in-domain data augmentation, speaker adaptation, prompt-based context mod- eling, target denoising fine-tuning (Meng et al., 2020), and model ensemble strategies. This task aims to build machine translation systems to trans- late conversational text and thus supports fluent communication between an agent speaking in En- ∗Work was done when Yunlong was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. †Jinan Xu is the corresponding author. 1The COMET is the official automatic evaluation metric. glish and a customer speaking in a different lan- guage (e.g., German), which is different from the first pre-training stage (Farajian et al., 2020; Liang et al., 2021a, 2022a; Liu et al., 2021; Gain et al., 2021, 2022; Buschbeck et al., 2022). Therefore, we mainly pay attention to the second fine-tuning stage. In the first pre-training stage, we follow pre- vious work (Meng et al., 2020; Zeng et al., 2021; Meng and Zhang, 2019; Yan et al., 2020) and utilize several effective Transformer vari- ants. Specifically, we combine the Multi-Head- Attention (Vaswani et al., 2017), Average Atten- tion Transformer (Zhang et al., 2018), and Talking- Heads Attention (Shazeer et al., 2020), which have shown significant model performance and diver- sity. For data augmentation, we employ the back- translation method to use the target-side mono- lingual data and apply the forward-translation to leverage the source-side monolingual data. To fully utilize the source-side of bilingual data, we use the sequence-level knowledge distillation method (Kim and Rush, 2016). In the second fine-tuning stage, for speaker- aware in-domain data augmentation, based on the BConTrasT …

Content truncated to 500 words