Samsung R&D Institute Poland submission to WMT20 News Translation Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 181–190 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 181 Samsung R&D Institute Poland submission to WMT20 News Translation Task Mateusz Krubi´nski, Marcin Chochowski, Bartłomiej Boczek, Mikołaj Koszowski, Adam Dobrowolski, Marcin Szyma´nski, Paweł Przybysz Samsung R&D Institute Poland Abstract This paper describes the submission to the WMT20 shared news translation task by Sam- sung R&D Institute Poland. We submitted systems for six language directions: English to Czech, Czech to English, English to Pol- ish, Polish to English, English to Inuktitut and Inuktitut to English. For each, we trained a single-direction model. However, directions including English, Polish and Czech were de- rived from a common multilingual base, which was later fine-tuned on each particular direc- tion. For all the translation directions, we used a similar training regime, with iterative training corpora improvement through back- translation and model ensembling. For the En → Cs direction, we additionally leveraged document-level information by re-ranking the beam output with a separate model. 1 Introduction Since the Transformer architecture became the stan- dard model in Neural Machine Translation, recent advancements in the field have come from two tech- niques. The first one is deepening the model by adding more layers, mainly in the encoder part, in order to model more complex dependencies (Ra- ganato and Tiedemann, 2018; Wang et al., 2019a; Wu et al., 2019). This, however, poses problems during the training – too deep models are much harder to train due to the gradient vanishing prob- lem (Zhang et al., 2019). The second technique consists in improving the quality of training data by removing spurious translations (Koehn et al., 2019) and making the data easier to learn through the teacher-student methodology (Hinton et al., 2015; Kim and Rush, 2016; Tan et al., 2019). In this submission, we decided to leverage both techniques. We deepened the model with a lexical- shortcuts transformer modification. We also iter- atively improved the synthetic corpora by train- ing better and better translation models, back- translating and distilling the data in each step. The remainder of this paper is structured as fol- lows: Section 2 introduces the data used for train- ing, Section 3 shows baseline NMT models and our experiments. In Section 4 we describe our training regime and results. Section 5 is for conclusions. 2 Data 2.1 Data Filtering We used all the parallel data available in the con- strained settings. We filtered the parallel data with a two-step process. First, we used a simple heuristics for general clean-up: • remove pairs where any of the sentences is longer than 1500 characters • remove sentences with characters not in the Unicode range specific to a given language pair • remove pairs based on a length-ratio thresh- old. We then de-duplicated the data and used the fast- align1 tool to filter out pairs basing on the align- ment probability between the source and the target (Table 1). For monolingual data, we used only the general clean-up procedure. 2.2 Data …

Content truncated to 500 words