The AISP-SJTU Translation System for WMT 2022

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 310 - 317 December 7-8, 2022 ©2022 Association for Computational Linguistics The AISP-SJTU Translation System for WMT 2022 Guangfeng Liu1 Qinpei Zhu1 Xingyu Chen2 Renjie Feng1 Jianxin Ren1 Renshou Wu1 Qingliang Miao1 Rui Wang2 Kai Yu1,2 1AI Speech Co., Ltd., Suzhou, China 2Shanghai Jiao Tong University, Shanghai, China Abstract This paper describes AISP-SJTU’s partici- pation in WMT 2022 shared general MT task. In this shared task, we participated in four translation directions: English→Chinese, Chinese→English, English→Japanese and Japanese→English. Our systems are based on the Transformer architecture with sev- eral novel and effective variants, including network depth and internal structure. In our experiments, we employ data filtering, large-scale back-translation, knowledge distil- lation, forward-translation, iterative in-domain knowledge finetune and model ensemble. The constrained systems achieve 48.8, 29.7, 39.3 and 22.0 case-sensitive BLEU scores on EN→ZH, ZH→EN, EN→JA and JA→EN, re- spectively. 1 Introduction We participate in the WMT 2022 shared general MT task, including English↔Chinese(EN↔ZH) and English↔Japanese(EN↔JA). All of our sys- tems are built with constrained data sets. For model architectures, we exploit several Transformer variants including transformer-DLCL (Wang et al., 2019), transformer-ODE (Bei Li, 2021), transformer-RPR (Shaw et al., 2018), transformer-Coda (Zheng et al., 2021). In this year’s translation tasks, we mainly em- ploy data filtering (Zhou et al., 2021; Zeng et al., 2021), large-scale back-translation (Sennrich et al., 2015; Lample et al., 2017), knowledge distillation, forward-translation, in-domain knowledge finetune and model ensemble to improve the final model’s performance. For the synthetic data generation, we first ex- ploit large-scale back-translation (Sennrich et al., 2015) method to leverage the target-side monolin- gual data and the knowledge distillation (Kim and Rush, 2016) to leverage the source-side of bilingual data. To use the source-side monolingual data, we explore forward-translation by ensemble models to get general domain synthetic data.Furthermore, several data augmentation methods are applied to improve the model robustness, including different token-level noise and different sampling methods. We mainly use three training strategies in the training phase, including the warmup strategy (He et al., 2016) to adjust the learning rate in training, different sampling methods (Holtzman et al., 2019) and the Graduated Label Smoothing (Wang et al., 2020). In the fine-tuning stage, the test set is clustered into seven categories, and then use the TFIDF- Ngram algorithm (Ramos et al., 2003) to search for similar bilingual and monolingual data in all data according to these seven domains. The monolin- gual data is then generated using forward transla- tion to generate pseudo-data, and finally fine-tuned together with the searched bilingual data. We pay more attention to the differences be- tween different models in this year. We compute Self-BLEU (Zhu et al., 2018) from the translations of the models on the valid set to quantify the di- versity among different models. To be precise, we use the translation of one model as the hypothesis and the translations of other models as references to calculate an average BLEU score. A lower Self- BLEU means this model is more …

Content truncated to 500 words