KIT’s Submission to the IWSLT 2019 Shared Task on Text Translation

KIT’s Submission to the IWSLT 2019 Shared Task on Text Translation Felix Schneider, Alex Waibel Karlsruhe Institute of Technology felix.schneider@kit.edu, waibel@kit.edu Abstract In this paper, we describe KIT’s submission for the IWSLT 2019 shared task on text translation. Our system is based on the transformer model [1] using our in-house implemen- tation. We augment the available training data using back- translation and employ fine-tuning for the final model. For our best results, we used a 12-layer transformer-big config- uration, achieving state-of-the-art results on the WMT2018 test set. We also experiment with student-teacher models to improve performance of smaller models. 1. Introduction The performance of state-of-the-art NMT systems can often be difficult to reproduce. It is highly dependent not only on the amount and kind of training data as well as on a num- ber of subtle hyperparameter choices and implementation de- tails. The Karlsruhe Institute of Technology perticipated in the IWSLT 2019 shared task on English to Czech text translation. In this paper we describe our data preprocessing, the model architectures we have chosen, as well as attempt to give an exhaustive list of the implementation and training tricks that we used to achieve our final performance. We describe the data preprocessing in section 2, back- translation in section 3, training speedups in section 4.1 and finetuning in section 4.2. 2. Data Preprocessing The allowed training data for the task was the special MUST- C release containing TED talks as well as all WMT 2019 data. We made use of all allowed data, which is broken down in table 1. The allowed parallel data from WMT consists of Commoncrawl, CzEng (which makes up the vast majority of the parallel training data), Europarl, news commentrary and paracrawl. WMT also provides a large amount of monolingual data for Czech. We used backtranslation (section 3) to synthesize this data into parallel corpora, more than doubling the total amount of available training data. We performed very little data preprocessing: We first re- move noise, by fixing typical encoding errors (such as re- placing u+0080, the e character in cp1252, but a control character in utf-8, with u+20ac, the unicode representation Dataset sentence pairs (K) words (K) EN CS Commoncrawl 162 3 348 2 927 CzEng 57 065 618 423 543 184 Europarl 641 15 623 12 994 MUST-C 128 2 413 2 000 News-Commentary 240 5 166 4 610 Paracrawl 2 982 48 918 44 100 Newscrawl CS 2007-18 72 155 — 1 019 492 Total Parallel 61 218 693 891 609 815 Total 133 373 — 1 692 308 Table 1: Data Overview of the e character), and removing html tags which may be left in the data. Then we filtered the data, with the same method used in [2]: Filter any sentence pair where the Czech side does not contain any accented characters. We also filtered any sentence pair where source and target are the same or where one side is empty. This filtering removes about 8% of the training data. Then …

Content truncated to 500 words