To Optimize, or Not to Optimize, That Is the Question: TelU-KU Models for WMT21 Large-Scale Multilingual Machine Translation

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 387–397 November 10–11, 2021. ©2021 Association for Computational Linguistics 387 To optimize, or not to optimize, that is the question: TelU-KU models for WMT21 Large-Scale Multilingual Machine Translation Sari Dewi Budiwati1,2∗, Tirana Noor Fatyanosa1∗, Mahendra Data1,3, Dedy Rahman Wijaya2, Patrick Adolf Telnoni2, Arie Ardiyanti Suryani4, Agus Pratondo2, Masayoshi Aritsugi5 1Graduate School of Science and Technology, Kumamoto University, Japan 2School of Applied Science, Telkom University, Indonesia 3Faculty of Computer Science, Brawijaya University, Indonesia 4School of Informatics, Telkom University, Indonesia 5Faculty of Advanced Science and Technology, Kumamoto University, Japan {saridewi, fatyanosa, mahendra.data}@dbms.cs.kumamoto-u.ac.jp {saridewi, dedyrw, patrickadolf, ardiyanti, pratondo}@telkomuniversity.ac.id mahendra.data@ub.ac.id, aritsugi@cs.kumamoto-u.ac.jp Abstract We describe TelU-KU models of large- scale multilingual machine translation for five Southeast Asian languages: Javanese, Indonesian, Malay, Tagalog, Tamil, and English. We explore a variation of hy- perparameters of flores101_mm100_175M model using random search with 10% of datasets to improve BLEU scores of all thirty language pairs. We submitted two models, TelU-KU-175M and TelU-KU- 175M_HPO, with average BLEU scores of 12.46 and 13.19, respectively. Our models show improvement in most language pairs after optimizing the hyperparameters. We also identified three language pairs that ob- tained a BLEU score of more than 15 while using less than 70 sentences of the train- ing dataset: Indonesian-Tagalog, Tagalog- Indonesian, and Malay-Tagalog. 1 Introduction This paper describes our participation in the WMT21 shared task of large-scale multilingual machine translation. Specifically, we chose small track #2, which involves thirty language pairs, and used the neural machine translation (NMT) method. We call our models TelU-KU (Telkom University - Kumamoto University) as we use our university name in our submis- sions. ∗These authors contributed equally. Everyone con- tributed to writing this paper. NMT has been widely used in machine trans- lation research for many languages. Currently NMT has become the state of the art of ma- chine translation with a large number of paral- lel corpus (Bojar et al., 2017; Nakazawa et al., 2017; Chu and Wang, 2018; Sutskever et al., 2014). Meanwhile, for low resources cases, the NMT tends to give poor translation re- sults (Duh et al., 2013; Sennrich and Zhang, 2019; Zoph et al., 2016; Koehn and Knowles, 2017). In order to get better translation re- sults of NMT for low resources languages, some approaches are applied, such as using a large number of monolingual corpora (Artetxe et al., 2018a,b; Lample et al., 2018b,a), apply- ing transfer learning approach to share lexical and sentence level representation (Gu et al., 2018), using sub-word representation (Durrani et al., 2019), and hyperparameter optimiza- tion (HPO) (Sennrich and Zhang, 2019; Ru- bino et al., 2020). HPO is an important part of building an NMT system in many real-world applications. In other words, selecting effective hyperparam- eters is critical to building a strong NMT sys- tem. However, in many cases, hyperparam- eters are often set manually based on intu- ition and heuristics mechanisms, tedious and error-prone processes that can lead to unre- liable experimental results and poor perfor- mance of …

Content truncated to 500 words