The University of Edinburgh’s Bengali-Hindi Submissions to the WMT21 News Translation Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 180–186 November 10–11, 2021. ©2021 Association for Computational Linguistics 180 The University of Edinburgh’s Bengali-Hindi Submissions to the WMT21 News Translation Task Proyag Pal Alham Fikri Aji Pinzhen Chen Sukanta Sen School of Informatics, University of Edinburgh, Scotland {proyag.pal, a.fikri, pinzhen.chen, ssen}@ed.ac.uk Abstract We describe the University of Edinburgh’s Bengali↔Hindi constrained systems submit- ted to the WMT21 News Translation task. We submitted ensembles of Transformer models built with large-scale back-translation and fine- tuned on subsets of training data retrieved based on similarity to the target domain. For both translation directions, our submissions are among the best-performing constrained systems according to human evaluation. 1 Introduction We present the University of Edinburgh’s participa- tion in the WMT21 news translation shared task on the Bengali→Hindi (Bn→Hi) and Hindi→Bengali (Hi→Bn) language pairs. We followed the con- strained condition, i.e. only using the data provided by the organizers. The training data for these lan- guage pairs consisted of noisy crawled data, and was mostly out-of-domain with respect to the val- idation and test domain. Therefore, most of our efforts concentrated on fine-tuning models to adapt to the target domain. We also explore multiple back-translation methods, and ensembles of mod- els trained and fine-tuned with different methods. Building our systems consisted of the following steps, each of which is described in more detail in the remaining sections of this paper: • Cleaning the noisy parallel data (Section 3). • Training ensembles of Transformer models on the cleaned provided data for back-translation; and using the back-translated data along with the clean parallel data to train new models (Section 4). • Fine-tuning the models on subsets of training data retrieved that are similar to the target do- main, based on different similarity measures (Section 5). • Ensembling various models and decoding with optimal parameters (Section 6). We also report some methods that we tried to use but did not work in Section 8. 2 Model Configuration Our models follow the Transformer-Big architec- ture (Vaswani et al., 2017): 6 layers of encoders and decoders, 16 heads, an embedding size of 1024, a unit size of 4096, etc. We found that smaller Transformer architectures performed worse. All models are trained with the same vocabu- lary of 32k SentencePiece subwords (Kudo and Richardson, 2018) to allow ensembling. We use a shared vocabulary between source and target, as well as tied embeddings (Press and Wolf, 2017). We tried other vocabulary sizes too: 5k, 10k, and 20k, though all of them had similar performance. We also included several special tokens in the vo- cabulary, of which we finally used only one for tagged back-translation (Caswell et al., 2019). We train models with 32GB dynamic batch size and an optimizer delay (Bogoychev et al., 2018) of 3 with the Adam optimizer (Kingma and Ba, 2015) under a learning rate of 0.0003, until we see no improvement within 10 consecutive validation steps. All models were trained with the Marian NMT toolkit (Junczys-Dowmunt et al., 2018)1 …

Content truncated to 500 words