Summer: WeChat Neural Machine Translation Systems for the WMT22 Biomedical Translation Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 920 - 924 December 7-8, 2022 ©2022 Association for Computational Linguistics Summer: WeChat Neural Machine Translation Systems for the WMT22 Biomedical Translation Task Ernan Li, Fandong Meng and and Jie Zhou Pattern Recognition Center, WeChat AI, Tencent Inc, China {cardli,fandongmeng,withtomzhou}@tencent.com Abstract This paper introduces WeChat’s participation in WMT 2022 shared biomedical translation task on Chinese→English. Our systems are based on the Transformer(Vaswani et al., 2017), and use several different Transformer struc- tures to improve the quality of translation. In our experiments, we employ data filter- ing, data generation, several variants of Trans- former, fine-tuning and model ensemble. Our Chinese→English system, named Summer, achieves the highest BLEU score among all submissions. 1 Introduction This article describes the WeChat’s participation in WMT 2022 shared biomedical translation task on Chinese→English. We improve the translation quality of the system by increasing the diversity of model structure and data, fine-tuning the model with in-domain data, inserting tags at the beginning of each source sentence and selecting models with high diversity and good performance for ensemble. For model architectures, our system adopt BIG and DEEP Transformer models which contain 10- layer and 20-layer encoders, 10240 and 4096 filter sizes, respectively, with TRANSFORMER-BIG setting (Vaswani et al., 2017). In order to increase the diversity of the model, we use structures such as Average Attention Transformer (AAN) (Zhang et al., 2018) and Mixed-AAN Transformer archi- tecture (Zeng et al., 2021) in the decoder part. For data generation, we use back- translation (Sennrich et al., 2016a), knowledge distillation (Kim and Rush, 2016), and forward- translation (Zeng et al., 2021) to improve data quality. And we use some data augmentation methods to improve the model robustness, such as adding synthetic noise and dynamic top-p sampling (Zeng et al., 2021). Furthermore, according to the different sources of the corpora, we add tags at the beginning of the source sentence to perform domain adaptation. For fine-tuning, we use in-domain bilingual cor- pus to fine-tune models from the general domain to the biomedical domain, and use target denois- ing (Meng et al., 2020) to improve the diversity of models and mitigate training-generation discrep- ancy. For model ensemble, we use Self-BLEU (Zhu et al., 2018) to evaluate the similarity between mod- els. We take the prediction of one model as the reference and use the prediction of the other model to calculate the BLEU score. The higher the Self- BLEU score, the lower the diversity of the models. In the remainder of this paper, we start with presenting the data strategy in Section 2. Then we describe our system details in Section 3. Section 4 presents the experimental results. Finally, we conclude our work in Section 5. 2 Data In this section, we introduce the details of bilingual and monolingual data used in this shared task. 2.1 Bilingual Corpus Our baseline model is trained with out-of-domain (OOD) data from WMT 2022 shared task on general machine translation1. Additionally, we use in-house data (depicted in …

Content truncated to 500 words