Towards Bidirectional Hierarchical Representations for Attention-based Neural Machine Translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1432–1441 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Towards Bidirectional Hierarchical Representations for Attention-Based Neural Machine Translation Baosong Yang† Derek F. Wong†∗ Tong Xiao‡ Lidia S. Chao† Jingbo Zhu‡ †NLP2CT Lab, Department of Computer and Information Science, University of Macau, Macau, China ‡NiuTrans Lab, Northeastern University, Shenyang, China nlp2ct.baosong@gmail.com, {derekfw,lidiasc}@umac.mo, {xiaotong,zhujingbo}@mail.neu.edu.cn Abstract This paper proposes a hierarchical atten- tional neural translation model which fo- cuses on enhancing source-side hierarchi- cal representations by covering both lo- cal and global semantic information us- ing a bidirectional tree-based encoder. To maximize the predictive likelihood of tar- get words, a weighted variant of an at- tention mechanism is used to balance the attentive information between lexical and phrase vectors. Using a tree-based rare word encoding, the proposed model is extended to sub-word level to allevi- ate the out-of-vocabulary (OOV) prob- lem. Empirical results reveal that the proposed model significantly outperforms sequence-to-sequence attention-based and tree-based neural translation models in English-Chinese translation tasks. 1 Introduction Neural machine translation (NMT) automatically learns the abstract features of and semantic re- lationship between the source and target sen- tences, and has recently given state-of-the-art re- sults for various translation tasks (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bah- danau et al., 2015). The most widely used model is the encoder-decoder framework (Sutskever et al., 2014), in which the source sentence is encoded into a dense representation, followed by a decod- ing process which generates the target translation. By exploiting the attention mechanism (Bahdanau et al., 2015), the generation of target words is con- ditional on the source hidden states, rather than on the context vector alone. From a model archi- tecture perspective, prior studies of the attentive ∗Corresponding author S0 PRP1 I VP2 VP3 VBP5 take PRT6 up NP4 NP7 DT9 a NN10 position PP8 IN11 in NP12 the room Figure 1: Induction of phrase and sentence rep- resentations over the syntactic structure of a sen- tence. encoder-decoder translation model are mainly di- vided into two types. The sequence-to-sequence model treats a sen- tence as a sequence of tokens. The most funda- mental approaches transform the source sentence sequentially into a fixed-length context vector, and the annotation vector of each word summa- rizes the preceding words (Sutskever et al., 2014; Cho et al., 2014b). Although Bahdanau et al. (2015) used a bidirectional recurrent neural net- work (RNN) (Schuster and Paliwal, 1997) to con- sider preceding and following words jointly, these sequential representations are insufficient to fully capture the semantics of a sentence, due to the fact that they do not account for the syntactic interpre- tations of sentence structure (Eriguchi et al., 2016; Tai et al., 2015). By incorporating additional fea- tures into a sequential model, Sennrich and Had- dow (2016) and Stahlberg et al. (2016) suggest that a greater amount of linguistic information can im- prove the translation performance. The tree-to-sequence model encodes a source sentence according to a given syntactic …

Content truncated to 500 words