Multilingual Neural Machine Translation with Task-Specific Attention

Proceedings of the 27th International Conference on Computational Linguistics, pages 3112–3122 Santa Fe, New Mexico, USA, August 20-26, 2018. 3112 Multilingual Neural Machine Translation with Task-Specific Attention Graeme Blackwood Miguel Ballesteros Todd Ward IBM Research AI 1101 Kitchawan Rd, Yorktown Heights, NY 10598, USA {blackwood,toddward}@us.ibm.com, miguel.ballesteros@ibm.com Abstract Multilingual machine translation addresses the task of translating between multiple source and target languages. We propose task-specific attention models, a simple but effective technique for improving the quality of sequence-to-sequence neural multilingual translation. Our approach seeks to retain as much of the parameter sharing generalization of NMT models as possible, while still allowing for language-specific specialization of the attention model to a particular language-pair or task. Our experiments on four languages of the Europarl corpus show that using a target-specific model of attention provides consistent gains in translation quality for all possible translation directions, compared to a model in which all parameters are shared. We observe improved translation quality even in the (extreme) low-resource zero-shot translation directions for which the model never saw explicitly paired parallel data. 1 Introduction and Motivation Multilingual machine translation is the task of building a system capable of translating between more than just a single source and target language. The approach is motivated by the idea that learning to translate between one pair of languages can help to improve the translation quality of related language pairs. Multilingual MT can be divided into three main types according to the support for source and target languages: (i) single source, multiple target (ii) multiple source, single target, and (iii) multiple source, multiple target. Our work focuses on improving the quality of fully n-way multilingual NMT models that can translate between multiple source and target languages. Multinational organizations and companies increasingly publish content in a variety of languages. In addition to exploiting multiple sources of parallel data and leveraging similarities across different lan- guages, multilingual translation can considerably simplify deployment. Directly supporting all possible translation directions for a set of n languages would require n(n − 1) sets of parallel data and trained models. This is an expensive proposal, from both a data and deployment perspective. The number of required systems can be reduced to n if translation by bridging is used. In contrast, a fully n-way mul- tilingual system can be trained to translate between all known language pairs using a single model. It is even possible to translate between language pairs that were never explicitly paired in the parallel training data, so called zero-shot translation (Johnson et al., 2017). Neural machine translation (NMT) (Bahdanau et al., 2014; Sutskever et al., 2014) has recently become the standard model of translation due to the high levels of fluency and accuracy that it can achieve (Koehn and Knowles, 2017). NMT is an excellent choice for multilingual translation since the neural architecture is language-agnostic and capable of capturing translation properties, such as long-distance re-ordering, between even highly dissimilar languages. One of the main advantages of NMT is that the model is able to learn …

Content truncated to 500 words