Montreal Neural Machine Translation Systems for WMT’15

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Montreal Neural Machine Translation Systems for WMT’15 S´ebastien Jean⋆ University of Montreal jeasebas@iro.umontreal.ca Orhan Firat⋆ Middle East Technical University, Turkey orhan.firat@ceng.metu.edu.tr Kyunghyun Cho University of Montreal Roland Memisevic University of Montreal firstname.lastname@umontreal.ca Yoshua Bengio University of Montreal CIFAR Senior Fellow Abstract Neural machine translation (NMT) systems have recently achieved re- sults comparable to the state of the art on a few translation tasks, including English→French and English→German. The main purpose of the Montreal In- stitute for Learning Algorithms (MILA) submission to WMT’15 is to evaluate this new approach on a greater variety of language pairs. Furthermore, the human evaluation campaign may help us and the research community to better understand the behaviour of our systems. We use the RNNsearch architecture, which adds an attention mechanism to the encoder- decoder. We also leverage some of the recent developments in NMT, including the use of large vocabularies, unknown word replacement and, to a limited degree, the inclusion of monolingual language models. 1 Introduction Neural machine translation (NMT) is a recently proposed approach for machine translation that re- lies only on neural networks. The NMT system is trained end-to-end to maximize the conditional probability of a correct translation given a source sentence (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015). Although NMT has only recently been introduced, its performance has been found to be comparable to the state-of-the-art statistical ma- chine translation (SMT) systems on a number of translation tasks (Luong et al., 2015; Jean et al., 2015). The main purpose of our submission to WMT’15 is to test the NMT system on a greater ⋆ equal contribution variety of language pairs. As such, we trained sys- tems on Czech↔English, German↔English and Finnish→English. Furthermore, the human evalu- ation campaign of WMT’15 will help us better un- derstand the quality of NMT systems which have mainly been evaluated using the automatic evalu- ation metric such as BLEU (Papineni et al., 2002). Most NMT systems are based on the encoder- decoder architecture (Cho et al., 2014; Sutskever et al., 2014; Kalchbrenner and Blunsom, 2013). The source sentence is first read by the encoder, which compresses it into a real-valued vector. From this vector representation the decoder may then generate a translation word-by-word. One limitation of this approach is that a source sen- tence of any length must be encoded into a fixed- length vector. To address this issue, our systems for WMT’15 use the RNNsearch architecture from (Bahdanau et al., 2015). In this case, the encoder assigns a context-dependent vector, or annotation, to every source word. The decoder then selectively combines the most relevant annotations to gener- ate each target word. NMT systems often use a limited vocabu- lary of approximately 30, 000 to 80, 000 target words, which leads them to generate many out- of-vocabulary tokens (⟨UNK⟩). This may easily lead to the degraded quality …

Content truncated to 500 words