Samsung and University of Edinburgh’s System for the IWSLT 2019

Samsung and University of Edinburgh’s System for the IWSLT 2019 Joanna Wetesko1, Marcin Chochowski1, Pawel Przybysz1, Philip Williams2, Roman Grundkiewicz2 Rico Sennrich2, Barry Haddow2, Antonio Valerio Miceli Barone2 Alexandra Birch2 1Samsung R&D Institute, Poland 2School of Informatics, University of Edinburgh {j.wetesko,m.chochowski,p.przybysz}@samsung.com {pwillia4,rgrundki,bhaddow,amiceli}@inf.ed.ac.uk, {rico.sennrich,a.birch}@ed.ac.uk Abstract This paper describes the joint submission to the IWSLT 2019 English to Czech task by Samsung R&D Institute, Poland, and the University of Edinburgh. Our submission was ul- timately produced by combining four Transformer systems through a mixture of ensembling and reranking. 1. Introduction This paper describes the joint submission to the IWSLT2019 Text Translation task by Samsung R&D Institute, Poland (SRPOL), and the University of Edinburgh (UEDIN). Our methods are based on our approach from last year [1] in- cluding exploiting non-parallel resources, large scale use of back-translation for model fine-tuning, as well as model en- sembling. This year, however, we decided to train our NMT systems using Marian toolkit [2], focusing on experimenting with bigger Transformer [3] architectures and significantly simplifying the data pre-processing stage. Our final submission involved the combination of four individual systems. Two of these were trained from scratch using a mixture of WMT and MUST-C data. We describe the training data in detail in Section 2 and the specifics of the system architecture and training in Section 3. Since we had already trained systems for the WMT19 News Translation Task, we experimented with fine-tuning on the MUST-C data and used the resulting systems for reranking. We describe these additional systems in Section 4. The final combination of systems used in our submission is detailed in Section 5. 2. Training Data 2.1. Provided training data We constructed several systems using different combina- tions of supplied data for training. We used the in-domain TED MUST-C data, as well as all other provided parallel resources. The amount of in-domain data was relatively small compared to the size of out-of-domain corpora. There- fore, we decided to generate more pseudo in-domain syn- thetic sentence pairs through back-translation of monolin- gual Czech CommonCrawl and NewsCrawl 2018, resulting in 32M additional parallel sentence pairs. The details of the back-translation system and the whole filtering process are described in section 2.2. As a result, we created three training sets used in our ex- periments. The main one was a set constructed using pro- vided parallel data, including 10 times ovesampled MUST-C TED sentence pairs (around 64M lines in total). This will be referred later as the base training set. It was further extended with back-translated lines from CommonCrawl (80M), con- stituting the second set. The third set was composed of syn- thetic back-translated sentence pairs from NewsCrawl (16M) and used only for further model domain adaptation. We noticed overlapping sentences in provided training and MUST-C development sets. After filtering them out from the development set, we got a set containing 975 lines. Dur- ing all trainings we used that pruned MUST-C development set for progress validation. We filtered out noisy lines in our training corpus, con- taining garbled …

Content truncated to 500 words