CUNI Basque-to-English Submission in IWSLT18

CUNI Basque-to-English Submission in IWSLT18 Tom Kocmi Duˇsan Variˇs Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostransk´e n´amˇest´ı 25, 118 00 Prague, Czech Republic <surname>@ufal.mff.cuni.cz Abstract We present our submission to the IWSLT18 Low Re- source task focused on the translation from Basque-to- English. Our submission is based on the current state- of-the-art self-attentive neural network architecture, Trans- former. We further improve this strong baseline by exploiting available monolingual data using the back-translation tech- nique. We also present further improvements gained by a transfer learning, a technique that trains a model using a high-resource language pair (Czech-English) and then fine- tunes the model using the target low-resource language pair (Basque-English). 1. Introduction Despite becoming the current dominant approach in the field of machine translation (MT), neural machine translation (NMT) [1] systems still perform poorly in certain scenar- ios. One of them is learning to translate between language pairs where only a small amount of parallel data is avail- able. Under these circumstances the NMT model quickly overfits and its performance plummets when translating sen- tences not seen during training. As observed in [2], with small parallel data, NMT performs much worse than the pre- vious approach of phrase-based MT. There are situations where the ability to learn an MT model of a reasonable quality given only a small amount of training data can be crucial. For example, when a crisis oc- curs in a region where an under-resourced language is spo- ken, a quick deployment of an MT system translating from or to that language can make a huge difference in the impact of the provided support [3]. In this paper, we describe the CUNI submission to the IWSLT Low Resource task for translating from Basque-to- English in the domain of TED talks. Our submission is based on the recently introduced self-attentive network architecture called Transformer [4]. We improve the performance of this model by exploiting the English in-domain monolingual data using the back-translation technique [5]. We achieve further improvements via transfer learning. Transfer learning [6, 7] consists of training a “parent” (high-resource) model first and then continuing the training on the “child”, low-resource, parallel data as a means of model adaptation. Furthermore, we combine several models saved at training checkpoints by simply averaging the weights (“model averaging”) as a sub- stitute of model ensembling. The structure of the paper is the following. In Section 2, we describe the method of transfer learning followed by the description of back-translation in Section 3. The model de- scription is presented in Section 4 and the dataset overview in Section 5. Section 6 details the results achieved by our systems. Section 7 discusses other works in the area of low-resource translation systems. And finally Section 8 con- cludes the paper. 2. Transfer learning Transfer learning is based on the observation that neural ma- chine translation model that is first trained on the parallel data of a high-resource language pair can be adapted to a lower-resource …

Content truncated to 500 words