The LIG system for the English-Czech Text Translation Task of IWSLT 2019

The LIG system for the English-Czech Text Translation Task of IWSLT 2019 Loïc Vial Benjamin Lecouteux Didier Schwab Hang Le Laurent Besacier Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France {loic.vial, benjamin.lecouteux, didier.schwab, thi-phuong-hang.le, laurent.besacier}@univ-grenoble-alpes.fr Abstract In this paper, we present our submission for the English to Czech Text Translation Task of IWSLT 2019. Our system aims to study how pre-trained language models, used as input embeddings, can improve a specialized machine translation system trained on few data. Therefore, we implemented a Transformer-based encoder- decoder neural system which is able to use the output of a pre-trained language model as input embeddings, and we com- pared its performance under three configurations: 1) without any pre-trained language model (constrained), 2) using a lan- guage model trained on the monolingual parts of the allowed English-Czech data (constrained), and 3) using a language model trained on a large quantity of external monolingual data (unconstrained). We used BERT as external pre-trained language model (configuration 3), and BERT architecture for training our own language model (configuration 2). Regarding the training data, we trained our MT system on a small quantity of parallel text: one set only consists of the provided MuST-C corpus, and the other set consists of the MuST-C corpus and the News Commentary corpus from WMT. We observed that using the external pre-trained BERT improves the scores of our system by +0.8 to +1.5 of BLEU on our development set, and +0.97 to +1.94 of BLEU on the test set. However, using our own language model trained only on the allowed parallel data seems to improve the machine translation performances only when the system is trained on the smallest dataset. 1. Introduction The recent advances in pre-trained Language Models [1, 2, 3, 4, 5] have shown that they could greatly improve many NLP tasks such as Natural Language Understanding, Ques- tion Answering, Natural Language Inference, Word Sense Disambiguation, etc. With our submission, we would like to explore what these models can bring to a typical Transformer-based encoder- decoder Neural Machine Translation system. Unlike the works of [6] and [3] where the authors fine-tune the weights of the language models on the translation task, we propose to use the language models as input embeddings for our neural system. We expect that the language model, because it is trained on a great quantity of monolingual data, will bring some additional information to a MT system trained on relatively few parallel data. Therefore, we conducted experiments that compare our system with and without the information from a BERT pre- trained model [2]. In addition, we created our own BERT LM by training it on the allowed training data only, in order to see if the language model is still useful in a constrained setting. For the training data, we used only the provided MuST-C [7] and News Commentary [8] from WMT, for a total of less than 400k parallel sentences. 2. System Description 2.1. Architecture Our system relies mostly on the Transformer architecture [9]. …

Content truncated to 500 words