CUNI English-Czech and English-Polish Systems in WMT20: Robust Document-Level Training

Proceedings of the 5th Conference on Machine Translation (WMT), pages 269–273 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 269 CUNI English-Czech and English-Polish Systems in WMT20: Robust Document-Level Training Martin Popel Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské námˇestí 25, 118 00 Prague, Czech Republic popel@ufal.mff.cuni.cz Abstract We describe our two NMT systems sub- mitted to the WMT 2020 shared task in English↔Czech and English↔Polish news translation. One system is sentence level, translating each sentence independently. The second system is document level, translating multiple sentences, trained on multi-sentence sequences up to 3000 characters long. 1 Introduction In this paper, we describe our two NMT systems submitted to the WMT 2020 news translation shared task: “CUNI-Transformer” (Charles University Transformer, sentence-level) and “CUNI-DocTransformer” (document-level). We trained them for English↔Czech and the for- mer one also for English↔Polish (no parallel document-level data was provided for English- Polish, thus we could not train the latter one). 2 Common settings Both our systems are implemented in the Ten- sor2Tensor framework (Vaswani et al., 2018) and have the same Transformer (Vaswani et al., 2017) architecture – transformer_big with 12 encoder lay- ers instead of the default 6 (while keeping 6 layers in the decoder). The 32k joint English-Czech sub- word vocabulary is exactly the same as used by Popel (2018) and Popel et al. (2019), which are the systems we submitted to WMT in the last two years. Also most of the hyperparameters (except for the encoder depth) and the training regime are the same. The main improvement of our sentence-level sys- tem relative to our last-year submission stems from using slightly larger and better-filtered training data – CzEng 2.0 (Kocmi et al., 2020b) with 61M authen- tic parallel and 127M synthetic (back-translated) sentence words (M) data set pairs (M) EN CS authentic 61 617 702 EN-mono (NewsCrawl 2016–2018) 76 1296 1474 CS-mono (NewsCrawl 2013–2018) 51 700 833 total 188 2613 3009 Table 1: Training data sizes (in millions). All the data are taken from CzEng 2.0. sentences (see Table 1), instead of CzEng 1.7 with 57M authentic parallel sentences. We also enlarged our development-test set: we concatenated WMT newstest 2008–2018, instead of using newstest2016 only. WMT news tests before 2020 did not have paragraph boundaries marked. We thus prepared a version of our dev- set where we joined together several consecutive sentences randomly (except for titles not ended by a punctuation) to simulate WMT2020 paragraph- level setting. Our document-level system was further im- proved as described in Sections 3 and 4. 3 Document-level training Our last-year document-level submission (Popel et al., 2019) introduced a method of training-data context augmentation, where multiple consecutive sentences (within original documents) are merged together into multi-sentence sequences (of parallel source-target data). The sentences within each se- quence are separated with a special token, so that we can easily extract the sentence alignment after decoding. The length of the sequences was lim- ited by 1000 characters and 200 subwords (i.e. any sequences …

Content truncated to 500 words