UdS-DFKI@WMT20: Unsupervised MT and Very Low Resource Supervised MT for German-Upper Sorbian

Proceedings of the 5th Conference on Machine Translation (WMT), pages 145–155 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 145 Tohoku-AIP-NTT at WMT 2020 News Translation Task Shun Kiyono ♠♦ Takumi Ito ♦♠ Ryuto Konno ♦♠ Makoto Morishita ♥♦ Jun Suzuki ♦♠∗ ♠RIKEN Center for Advanced Intelligence Project ♦Tohoku University ♥NTT Communication Science Laboratories shun.kiyono@riken.jp; {t-ito, ryuto, jun.suzuki}@ecei.tohoku.ac.jp; makoto.morishita.gr@hco.ntt.co.jp Abstract In this paper, we describe the submis- sion of Tohoku-AIP-NTT to the WMT’20 news translation task. We participated in this task in two language pairs and four language directions: English↔German and English↔Japanese. Our system consists of techniques such as back-translation and fine- tuning, which are already widely adopted in translation tasks. We attempted to develop new methods for both synthetic data filtering and reranking. However, the methods turned out to be ineffective, and they provided us with no significant improvement over the baseline. We analyze these negative results to provide insights for future studies. 1 Introduction The joint team of Tohoku University, RIKEN AIP, and NTT (Tohoku-AIP-NTT) participated in the WMT’20 shared news translation task in two language pairs and four language directions: English→German (En→De), German→English (De→En), English→Japanese (En→Ja), and Japanese→English (Ja→En). At the very beginning of this year’s shared task, we planned to employ the following two enhance- ments at the core of our system. The first enhance- ment is the noisy synthetic data filtering (Koehn et al., 2018) to better utilize the millions of back- translated synthetic data. However, as we analyze in Section 5.1, this filtering turned out to be in- effective. The second enhancement is the rerank- ing of n-best candidates generated a the model. ∗ Shun conducted most of the experiments for both En↔De and En↔Ja. Takumi preprocessed En↔Ja data. Ryuto trained fasttext word vectors and implemented the post- ensemble method. Takumi and Ryuto worked on synthetic data filtering approaches. Makoto back-translated monolin- gual corpus for all language directions. Shun, Makoto, and Takumi developed an effective fine-tuning strategy. Jun imple- mented the entire reranking module and organized the team. Everyone contributed to writing this paper. Target Monolingual Target-to-Source Model Bitext Synthetic Data Bitext In-domain Bitext Test Data Source-to-Target Model N-best Candidates Translated Texts Left-to-Right Source-to-Target Model Right-to-Left Source-to-Target Model Left-to-Right Target-to-Source Model Left-to-Right Target-to-Source Model Masked Language Model ×2 Uni-directional Language Model ×2 Final Output Base Model (Section 3.1) Large-scale Back-translation (Section 3.3) Fine-tuning (Section 3.4) Ensemble (Section 3.5) Right-to-Left Models (Section 3.6) Reranking (Section 3.7) Post-ensemble (Section 3.9) Figure 1: Overview of our system. Given a collection of scores from multiple gen- erative/translation models, our reranking module selects the best candidate. We attempted to develop sophisticated machine learning based methods for optimizing the weight of each score. However, we found that those methods are not as effective as the simple grid search on the BLEU score (details in Section 3.7 and Section 5.3). Eventually, we designed our system as a com- bination of techniques that are already widely adopted in the shared task, such as back-translation and fine-tuning. The overview of our system …

Content truncated to 500 words