Understanding and Improving Hidden Representations for Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 466–477 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 466 Understanding and Improving Hidden Representation for Neural Machine Translation Guanlin Liϵ, Lemao Liuλ, Xintong Li∞, Conghui Zhuϵ ∗, Tiejun Zhaoϵ, Shuming Shiλ ϵHarbin Institute of Technology, λ Tencent AI Lab ∞ The Chinese University of Hong Kong {epsilonlee.green, znculee}@gmail.com, {chzhu, tjzhao}@hit.edu.cn, {redmondliu, shumingshi}@tencent.com Abstract Multilayer architectures are currently the gold standard for large-scale neural machine trans- lation. Existing works have explored some methods for understanding the hidden repre- sentations, however, they have not sought to improve the translation quality rationally ac- cording to their understanding. Towards un- derstanding for performance improvement, we first artificially construct a sequence of nested relative tasks and measure the feature gen- eralization ability of the learned hidden rep- resentation over these tasks. Based on our understanding, we then propose to regularize the layer-wise representations with all tree- induced tasks. To overcome the computa- tional bottleneck resulting from the large num- ber of regularization terms, we design efficient approximation methods by selecting a few coarse-to-fine tasks for regularization. Exten- sive experiments on two widely-used datasets demonstrate the proposed methods only lead to small extra overheads in training but no ad- ditional overheads in testing, and achieve con- sistent improvements (up to +1.3 BLEU) com- pared to the state-of-the-art translation model. 1 Introduction Neural machine translation (NMT) has witnessed great successes in recent years (Bahdanau et al., 2014; Wu et al., 2016). Current state-of-the-art (SOTA) NMT models are mainly constructed by a stacked neural architecture consisting of multiple hidden layers from bottom-up, where a classifier is built upon the topmost layer to solve the target task of translation (Gehring et al., 2017; Vaswani et al., 2017). Most works tend to focus on the transla- tion performance of the classifier defined on the topmost layer, however, they do not deeply under- stand the learned representations of hidden layers. Shi et al. (2016) and Belinkov et al. (2017) attempt ∗ Conghui Zhu is the corresponding author. root 0 1 00 01 10 11 000 001 010 011 110 111 x, y𝑡< y𝑡 is … 1100 1101 0100 0111 0000 0001 apple … cola … rice … cat … dog … wolf … run … of … in … 1 1 2 3 4 2 3 4 layer depth task HR Structural HR Figure 1: The structural hierarchical regularization framework. On the left is a 4-layer NMT decoder; on the right is a hierarchical clustering tree and the tree- induced relative tasks at every tree depth. to understand the hidden representations through the lens of a few linguistic tasks, while Ding et al. (2017) and Strobelt et al. (2018) propose appeal- ing visualization approaches to understand NMT models including the representation of hidden lay- ers. However, employing the analyses to moti- vate new methods for better translation, the ulti- mate goal of understanding NMT, is not achieved in these works. In our paper, we aim at understanding the hid- den …

Content truncated to 500 words