Understanding and Improving Morphological Learning in the Neural Machine Translation Decoder

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 142–151, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Understanding and Improving Morphological Learning in the Neural Machine Translation Decoder Fahim Dalvi Nadir Durrani Hassan Sajjad Yonatan Belinkov∗ Stephan Vogel Qatar Computing Research Institute – HBKU, Doha, Qatar {faimaduddin, ndurrani, hsajjad, svogel}@qf.org.qa ∗MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA 02139, USA belinkov@mit.edu Abstract End-to-end training makes the neural ma- chine translation (NMT) architecture sim- pler, yet elegant compared to traditional statistical machine translation (SMT). However, little is known about linguis- tic patterns of morphology, syntax and semantics learned during the training of NMT systems, and more importantly, which parts of the architecture are re- sponsible for learning each of these phe- nomena. In this paper we i) analyze how much morphology an NMT decoder learns, and ii) investigate whether inject- ing target morphology into the decoder helps it produce better translations. To this end we present three methods: i) joint generation, ii) joint-data learning, and iii) multi-task learning. Our results show that explicit morphological information helps the decoder learn target language mor- phology and improves the translation qual- ity by 0.2–0.6 BLEU points. 1 Introduction Neural machine translation (NMT) offers an el- egant end-to-end architecture, improving transla- tion quality compared to traditional phrase-based machine translation. These improvements are at- tributed to more fluent output (Toral and S´anchez- Cartagena, 2017) and better handling of mor- phology and long-range dependencies (Bentivogli et al., 2016). However, systematic studies are re- quired to understand what kinds of linguistic phe- nomena (morphology, syntax, semantics, etc.) are learned by these models and more importantly, which of the components is responsible for each phenomenon. A few attempts have been made to understand what NMT models learn about morphology (Be- linkov et al., 2017a), syntax (Shi et al., 2016) and semantics (Belinkov et al., 2017b). Shi et al. (2016) used activations at various layers from the NMT encoder to predict syntactic properties on the source-side, while Belinkov et al. (2017a) and Belinkov et al. (2017b) used a similar approach to investigate the quality of word representations on the task of morphological and semantic tagging. Belinkov et al. (2017a) found that word rep- resentations learned from the encoder are rich in morphological information, while representations learned from the decoder are significantly poorer. However, the paper does not present a convincing explanation for this finding. Our first contribution in this work is to provide a more comprehensive analysis of morphological learning on the decoder side. We hypothesize that other components of the NMT architecture – specifically the encoder and the attention mechanism, learn enough informa- tion about the target language morphology for the decoder to perform reasonably well, without in- corporating high levels of morphological knowl- edge into the decoder. To probe this hypothesis, we investigate the following questions: • What is the effect of attention on the perfor- mance of the decoder? • How much does the encoder help the decoder in …

Content truncated to 500 words