Incorporating Syntactic Uncertainty in Neural Machine Translation with a Forest-to-Sequence Model

Proceedings of the 27th International Conference on Computational Linguistics, pages 1421–1429 Santa Fe, New Mexico, USA, August 20-26, 2018. 1421 Incorporating Syntactic Uncertainty in Neural Machine Translation with a Forest-to-Sequence Model Poorya Zaremoodi Gholamreza Haffari Faculty of Information Technology, Monash University firstname.lastname@monash.edu Abstract Incorporating syntactic information in Neural Machine Translation (NMT) can lead to better reorderings, particularly useful when the language pairs are syntactically highly divergent or when the training bitext is not large. Previous work on using syntactic information, provided by top-1 parse trees generated by (inevitably error-prone) parsers, has been promising. In this paper, we propose a forest-to-sequence NMT model to make use of exponentially many parse trees of the source sentence to compensate for the parser errors. Our method represents the collection of parse trees as a packed forest, and learns a neural transducer to translate from the input forest to the target sentence. Experiments on English to German, Chinese and Farsi translation tasks show the superiority of our approach over the sequence-to-sequence and tree-to-sequence neural translation models. 1 Introduction The neural approach is revolutionising machine translation (MT). The main neural approach to MT is based on the encoder-decoder architecture (Cho et al., 2014; Sutskever et al., 2014), where an encoder (e.g a recurrent neural network) reads the source sentences sequentially to produce a fixed-length vector representation. Then, a decoder generates the translation from the encoded vector, which can dynami- cally change using the attention mechanism. One of the main premises about natural language is that words of a sentence are inter-related accord- ing to a (latent) hierarchical structure, i.e. a syntactic tree. Therefore, it is expected that modeling the syntactic structure should improve the performance of NMT, especially in low-resource or linguistically divergent scenarios, such as English-Farsi. In this direction, (Li et al., 2017) uses a sequence-to-sequence model, making use of linearised parse trees. (Chen et al., 2017b) has proposed a model which uses syntax to constrain the dynamic encoding of the source sentence via structurally constrained attention. (Bastings et al., 2017; Shuangzhi Wu, 2017) have incorporated syntactic information provided by the dependency tree of the source sentence. (Marcheggiani et al., 2018) has proposed a model to inject semantic bias into the encoder of NMT model. Recently, (Eriguchi et al., 2016; Chen et al., 2017a) have proposed methods to incorporate the hierar- chical syntactic constituency information of the source sentence. In addition to the embedding of words, computed using the vanilla sequential encoder, they compute the embeddings of phrases recursively, directed by the top-1 parse tree of the source sentence generated by a parser. Though the results are promising, the top-1 trees are prone to parser error, and furthermore cannot capture semantic ambiguities of the source sentence. In this paper, we address the aforementioned issues by using exponentially many trees encoded in a forest instead of a single top-1 parse tree. We capture the parser uncertainty by considering many parse trees and their probabilities. The encoding of each source sentence is guided by the forest, and …

Content truncated to 500 words