Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning

Proceedings of the Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 80–89 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning Jan Niehues and Eunah Cho Institute for Anthropomatics and Robotics KIT - Karlsruhe Institute of Technology, Germany firstname.lastname@kit.edu Abstract Linguistic resources such as part-of- speech (POS) tags have been extensively used in statistical machine translation (SMT) frameworks and have yielded bet- ter performances. However, usage of such linguistic annotations in neural machine translation (NMT) systems has been left under-explored. In this work, we show that multi-task learning is a successful and a easy ap- proach to introduce an additional knowl- edge into an end-to-end neural attentional model. By jointly training several natu- ral language processing (NLP) tasks in one system, we are able to leverage common information and improve the performance of the individual task. We analyze the impact of three design de- cisions in multi-task learning: the tasks used in training, the training schedule, and the degree of parameter sharing across the tasks, which is defined by the network ar- chitecture. The experiments are conducted for an German to English translation task. As additional linguistic resources, we ex- ploit POS information and named-entities (NE). Experiments show that the transla- tion quality can be improved by up to 1.5 BLEU points under the low-resource con- dition. The performance of the POS tag- ger is also improved using the multi-task learning scheme. 1 Introduction Recently, there has been a dramatic change in the state-of-the-art techniques for machine translation (MT). In a traditional method, often the best per- formance is achieved by using a complicated com- bination of several statistical models, which are in- dividually trained. For example, POS information was shown to be very helpful to model word re- ordering between languages, as shown in Niehues and Kolss (2009). While the recent development of end-to-end trained neural models (Bahdanau et al., 2014) showed significant gains over tradi- tional approaches, they are often trained only on the parallel data in an end-to-end fashion. In most cases, therefore, they do not facilitate other knowl- edge sources. When parallel data is sparse, exploiting other knowledge sources can be crucial for perfor- mance. Two techniques to integrate the additional resources are well studied. In one technique, we train a tool on the additional resources (e.g. POS tagger) and then annotate the parallel data using this tool. This technique has been applied exten- sively in SMT systems (e.g. Niehues and Kolss (2009)) as well as in some NMT systems (e.g. Sennrich and Haddow (2016)). The second tech- nique would be to use the annotated data directly to train the model. The goal of this work is to integrate the ad- ditional linguistic resources directly into neural models, in order to achieve better performance. To do so, we build a multi-task model and train sev- eral NLP tasks jointly. We use an attention-based sequence-to- sequence model for all tasks. Experiments show that we …

Content truncated to 500 words