NT5 at WMT 2022 General Translation Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 318 - 325 December 7-8, 2022 ©2022 Association for Computational Linguistics NT5 at WMT 2022 General Translation Task ∗Makoto Morishita ♠, ∗Keito Kudo ♢, ∗Yui Oka ♠, ∗Katsuki Chousa ♠, †Shun Kiyono ♡, †Sho Takase ♣, Jun Suzuki ♢♡ ♠NTT Communication Science Laboratories ♢Tohoku University ♡RIKEN Center for Advanced Intelligence Project ♣Tokyo Institute of Technology Abstract This paper describes the NTT-Tohoku- TokyoTech-RIKEN (NT5) team’s submission system for the WMT’22 general translation task. This year, we focused on the English-to- Japanese and Japanese-to-English translation tracks. Our submission system consists of an ensemble of Transformer models with several extensions. We also applied data augmentation and selection techniques to obtain potentially effective training data for training individual Transformer models in the pre-training and fine-tuning scheme. Additionally, we report our trial of incorporating a reranking module and the reevaluated results of several techniques that have been recently developed and published. 1 Introduction This paper describes an overview of our submission systems for participating in the WMT 2022 gen- eral machine translation tasks. Our team, named NT5, is comprised of individuals from four organi- zations: NTT, Tohoku University, Tokyo Institute of Technology, and RIKEN. This year, we focused on bi-directional translation in a single language pair: English-to-Japanese (En→Ja) and Japanese- to-English (Ja→En) translation tracks. Our submission system consists of an ensemble of Transformer models (Vaswani et al., 2017) with several recent extensions. We also applied data aug- mentation and selection techniques to obtain poten- ∗: Equal contributions. Morishita trained the initial trans- lation model and created a synthetic corpus. Morishita and Kudo tuned the model’s hyper-parameters. Oka implemented the relative positional embeddings. Chousa worked on the data filtering and the reranking module. Kiyono and Takase imple- mented the B2T connections and helped with the experiments. Morishita and Oka implemented the in-domain data selection methods for fine-tuning. Morishita, Kudo, and Suzuki devel- oped an effective fine-tuning strategy. Morishita, Kudo, and Suzuki trained the main translation models. Suzuki organized the team. Everyone contributed to writing this paper. †: Their current affiliation is LINE Corporation. Bitext Target Monolingual Translated Texts Target-to-Source Model Synthetic Data Bitext Selected data for Fine-tuning Test Set N-best Candidates … Source-to-Target Model Reranking Module Final Output In-domain Data Selection Module Source-to-Target Fine-tuned Model Figure 1: System overview tially effective training data for training individual Transformer models in the pre-training/fine-tuning scheme. The models were first trained with a large but possibly noisy parallel corpus for pre-training and then with a small but clean parallel corpus for fine-tuning. Additionally, we report our trial of incorporating a reranking module that rescores the n-best lists based on source-to-target, target-to- source, and masked language models. The following section briefly provides an overview of our entire system and each module in more depth. 2 System Overview Figure 1 shows an overview of our system. Our submissions are for the constrained track, which only uses parallel and monolingual data that are provided by the WMT shared-task organizers. 318 We selected Transformer …

Content truncated to 500 words