Towards Decoding as Continuous Optimisation in Neural Machine Translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 146–156 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Towards Decoding as Continuous Optimisation in Neural Machine Translation Cong Duy Vu Hoang† and Gholamreza Haffari‡ and Trevor Cohn† † University of Melbourne Melbourne, VIC, Australia ‡ Monash University Clayton, VIC, Australia vhoang2@student.unimelb.edu.au, gholamreza.haffari@monash.edu, t.cohn@unimelb.edu.au Abstract We propose a novel decoding approach for neural machine translation (NMT) based on continuous optimisation. We refor- mulate decoding, a discrete optimization problem, into a continuous problem, such that optimization can make use of efficient gradient-based techniques. Our powerful decoding framework allows for more ac- curate decoding for standard neural ma- chine translation models, as well as en- abling decoding in intractable models such as intersection of several different NMT models. Our empirical results show that our decoding framework is effective, and can leads to substantial improvements in translations, especially in situations where greedy search and beam search are not fea- sible. Finally, we show how the technique is highly competitive with, and comple- mentary to, reranking. 1 Introduction Sequence to sequence learning with neural net- works (Graves, 2013; Sutskever et al., 2014; Lip- ton et al., 2015) is typically associated with two phases: training and decoding (a.k.a. inference). Model parameters are learned by optimising the training objective, in order that the model can produce good translations when decoding unseen sentences. The majority of research has focused on the training paradigm or network architec- ture, however effective means of decoding have been under-investigated. Conventional heuristic- based approaches for approximate inference in- clude greedy, beam, and stochastic search. Greedy and beam search have been empirically proved to be adequate for many sequence to sequence tasks, and are the standard methods for NMT decoding. However, these inference approaches have sev- eral drawbacks. Firstly, although NMT models use a left-to-right generation which would appear to facilitate efficient search, the models themselves use a recurrent architecture, and accordingly are non-Markov. This prevents exact dynamic pro- gramming solutions, and moreover, limits the po- tential to incorporate additional global features or constraints. Global factors can be highly useful in producing better and more diverse translations. Secondly, the sequential decoding of symbols in the target sequence, the inter-dependencies among the target symbols are not fully exploited. For example, when decoding the words of the target sentence in a left-to-right manner, the right con- text is not exploited leading potentially to inferior performance (see Watanabe and Sumita (2002a) who apply this idea in traditional statistical MT). A natural way to capture this is to intersect left- to-right and right-to-left models, however the re- sulting model has no natural generation order, and thus standard decoding methods are unsuitable. We introduce a novel decoding framework (§ 3) that relaxes this discrete optimisation problem into a continuous optimisation problem. This is akin to linear programming relaxation approach for ap- proximate inference in graphical models with dis- crete random variables, where the exact inference is NP-hard (Sontag, 2010; Belanger and McCal- lum, …

Content truncated to 500 words