Latent Part-of-Speech Sequences for Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 780–790, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 780 Latent Part-of-Speech Sequences for Neural Machine Translation Xuewen Yang1, Yingru Liu1, Dongliang Xie2, Xin Wang1, and Niranjan Balasubramanian1,3 1Stony Brook University 2Beijing University of Posts and Telecommunications 1{xuewen.yang,yingru.liu,x.wang}@stonybrook.edu 2xiedl@bupt.edu.cn 3niranjan@cs.stonybrook.edu Abstract Learning target side syntactic structure has been shown to improve Neural Machine Trans- lation (NMT). However, incorporating syntax through latent variables introduces additional complexity in inference, as the models need to marginalize over the latent syntactic struc- tures. To avoid this, models often resort to greedy search which only allows them to ex- plore a limited portion of the latent space. In this work, we introduce a new latent vari- able model, LaSyn, that captures the co- dependence between syntax and semantics, while allowing for effective and efficient in- ference over the latent space. LaSyn decou- ples direct dependence between successive la- tent variables, which allows its decoder to ex- haustively search through the latent syntactic choices, while keeping decoding speed pro- portional to the size of the latent variable vo- cabulary. We implement LaSyn by modifying a transformer-based NMT system and design a neural expectation maximization algorithm that we regularize with part-of-speech infor- mation as the latent sequences. Evaluations on four different MT tasks show that incorpo- rating target side syntax with LaSyn improves both translation quality, and also provides an opportunity to improve diversity. 1 Introduction Syntactic information has been shown to im- prove the translation quality in NMT models. On the source side, syntax can be incorporated in multiple ways — either directly during encod- ing (Chen et al., 2018; Sennrich and Haddow, 2016; Eriguchi et al., 2016), or indirectly via multi-task learning to produce syntax informed representations (Eriguchi et al., 2017; Baniata et al., 2018; Niehues and Cho, 2017; Zaremoodi et al., 2018). On the target side, however, incor- porating the syntax is more challenging due to the additional complexity in inference when decoding over latent states. To avoid this, existing meth- ods resort to approximate inference over the latent states using a two-step decoding process (G¯u et al., 2018; Wang et al., 2018a; Wu et al., 2017; Aha- roni and Goldberg, 2017). Typically, the first stage decoder produces a beam of latent states, which serve as conditions to feed into the second stage decoder to obtain the target words. Thus, training and inference in these models can only explore a limited sub-space of the latent states. In this work, we introduce LaSyn, a new tar- get side syntax model that allows exhaustive ex- ploration of the latent states to ensure a better translation quality. Similar to prior work, LaSyn approximates the co-dependence between syntax and semantics of the target sentences by model- ing the joint conditional probability of the target words and the syntactic information at each po- sition. However, unlike prior work, LaSyn elim- inates the sequential …

Content truncated to 500 words