Visualizing and Understanding Neural Machine Translation

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1150–1159 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1106 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1150–1159 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1106 Visualizing and Understanding Neural Machine Translation Yanzhuo Ding† Yang Liu†‡∗Huanbo Luan† Maosong Sun†‡ †State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology, Tsinghua University, Beijing, China ‡Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China djx133@yeah.net, liuyang2011@tsinghua.edu.cn luanhuanbo@gmail.com, sms@tsinghua.edu.cn Abstract While neural machine translation (NMT) has made remarkable progress in recent years, it is hard to interpret its inter- nal workings due to the continuous rep- resentations and non-linearity of neural networks. In this work, we propose to use layer-wise relevance propagation (LRP) to compute the contribution of each contextual word to arbitrary hid- den states in the attention-based encoder- decoder framework. We show that visu- alization with LRP helps to interpret the internal workings of NMT and analyze translation errors. 1 Introduction End-to-end neural machine translation (NMT), which leverages neural networks to directly map between natural languages, has gained increasing popularity recently (Sutskever et al., 2014; Bah- danau et al., 2015). NMT proves to outperform conventional statistical machine translation (SMT) significantly across a variety of language pairs (Junczys-Dowmunt et al., 2016) and becomes the new de facto method in practical MT systems (Wu et al., 2016). However, there still remains a severe challenge: it is hard to interpret the internal workings of NMT. In SMT (Koehn et al., 2003; Chiang, 2005), the translation process can be denoted as a deriva- tion that comprises a sequence of translation rules (e.g., phrase pairs and synchronous CFG rules). Defined on language structures with varying gran- ularities, these translation rules are interpretable from a linguistic perspective. In contrast, NMT takes an end-to-end approach: all internal infor- mation is represented as real-valued vectors or ∗Corresponding author. matrices. It is challenging to associate hidden states in neural networks with interpretable lan- guage structures. As a result, the lack of inter- pretability makes it very difficult to understand translation process and debug NMT systems. Therefore, it is important to develop new meth- ods for visualizing and understanding NMT. Ex- isting work on visualizing and interpreting neu- ral models has been extensively investigated in computer vision (Krizhevsky et al., 2012; Mahen- dran and Vedaldi, 2015; Szegedy et al., 2014; Si- monyan et al., 2014; Nguyen et al., 2015; Girshick et al., 2014; Bach et al., 2015). Although visu- alizing and interpreting neural models for natural language processing has started to attract attention recently (Karpathy et al., 2016; Li et al., 2016), to the best of our knowledge, there is no exist- ing work on visualizing NMT models. Note that the attention mechanism (Bahdanau et al., 2015) is restricted to demonstrate the connection between words in source and target …

Content truncated to 500 words