Improving Neural Translation Models with Linguistic Factors

Improving Neural Translation Models with Linguistic Factors Cong Duy Vu Hoang University of Melbourne Melbourne, VIC, Australia vhoang2@student.unimelb.edu.au Gholamreza Haffari Monash University Clayton, VIC, Australia gholamreza.haffari@monash.edu Trevor Cohn University of Melbourne Melbourne, VIC, Australia t.cohn@unimelb.edu.au Abstract This paper presents an extension of neu- ral machine translation (NMT) model to incorporate additional word-level linguis- tic factors. Adding such linguistic fac- tors may be of great benefits to learn- ing of NMT models, potentially reduc- ing language ambiguity or alleviating data sparseness problem (Koehn and Hoang, 2007). We explore different linguistic annotations at the word level, includ- ing: lemmatization, word clusters, Part-of- Speech tags, and labeled dependency re- lations. We then propose different neural attention architectures to integrate these additional factors into the NMT frame- work. Evaluating on translating between English and German in two directions with a low resource setting in the domain of TED talks, we obtain promising re- sults in terms of both perplexity reductions and improved BLEU scores over baseline methods. 1 Introduction Neural Machine Translation (NMT) (Devlin et al., 2014; Bahdanau et al., 2015) is a new paradigm in machine translation (MT) powered by recent ad- vances in sequence to sequence learning frame- works (Graves, 2013; Sutskever et al., 2014). NMT has already made remarkable results and im- provements over conventional SMT (Luong et al., 2015). The core idea of NMT is the encoder-decoder framework where an encoder encodes the source sequence into a vector representation, and then a decoder generates the target sequence sequen- tially via a recurrent neural network (RNN). The use of a RNN provides the ability to memorize longer range dependencies that are impossible with standard n-gram modeling - a core compo- nent of the traditional Statistical Machine Transla- tion (SMT) framework (Koehn et al., 2003; Lopez, 2008; Koehn, 2010). Unlike the traditional SMT, NMT offers unique mechanisms to learn transla- tion equivalence without extensive feature engi- neering efforts. Though promising, NMT still lacks of the abil- ity of modeling deeper semantic and syntactic as- pects of the language. Koehn and Hoang (2007) presented a factored translation model to ad- dress this issue for the traditional SMT framework (Koehn et al., 2007), where the model incorporates various linguistic annotations for the surface level words. Particularly for low-resource conditions, these extra annotations can lead to better transla- tion of OOVs (or low-count words) and resolve ambiguities, hence increase the generalization ca- pabilities of the model. In machine translation with a low-resource set- ting, resolving data sparseness and semantic ambi- guity problems can help improve its performance. In this paper, we investigate utilizing extra syntac- tic and semantic linguistic factors in the context of the NMT framework. Linguistic factors can in- clude bundles of features, e.g., stems, roots, lem- mas, morphological classes, data-driven clusters, syntactic analyses (part-of-speeches, constituency parsing, dependency parsing). Adding such ex- tra factors may be of great benefits to NMT mod- els, potentially reducing language ambiguity and alleviating data sparseness further. In this paper, we explore four word-level factor annotations, in- cluding: …

Content truncated to 500 words