Modelling and Optimizing on Syntactic N-Grams for Statistical Machine Translation

Modelling and Optimizing on Syntactic N-Grams for Statistical Machine Translation Rico Sennrich School of Informatics University of Edinburgh 10 Crichton Street Edinburgh EH8 9AB Scotland, UK Abstract The role of language models in SMT is to promote fluent translation output, but tradi- tional n-gram language models are unable to capture fluency phenomena between distant words, such as some morphological agree- ment phenomena, subcategorisation, and syn- tactic collocations with string-level gaps. Syn- tactic language models have the potential to fill this modelling gap. We propose a lan- guage model for dependency structures that is relational rather than configurational and thus particularly suited for languages with a (rel- atively) free word order. It is trainable with Neural Networks, and not only improves over standard n-gram language models, but also outperforms related syntactic language mod- els. We empirically demonstrate its effective- ness in terms of perplexity and as a feature function in string-to-tree SMT from English to German and Russian. We also show that using a syntactic evaluation metric to tune the log-linear parameters of an SMT system fur- ther increases translation quality when cou- pled with a syntactic language model. 1 Introduction Many languages exhibit fluency phenomena that are discontinuous in the surface string, and are thus not modelled well by traditional n-gram language mod- els. Examples include morphological agreement, e.g. subject-verb agreement in languages that do not (exclusively) follow SVO word order, subcategori- sation, and collocations involving distant, but syn- tactically linked words. Syntactic language models try to overcome the limitation to a local n-gram context by using syn- tactically related words (and non-terminals) as con- text information. Despite their theoretical attractive- ness, it has proven difficult to improve SMT with parsers as language models (Och et al., 2004; Post and Gildea, 2008). This paper describes an effective method to model, train, decode with, and weight a syntactic language model for SMT. While all these aspects are important for successfully applying a syntactic lan- guage model, our primary contributions are a novel dependency language model which improves over prior work by making relational modelling assump- tions, which we argue are better suited for languages with a (relatively) free word order, and the use of a syntactic evaluation metric for optimizing the log- linear parameters of the SMT model. While language models that operate on words linked through a dependency chain – called syntactic n-grams (Sidorov et al., 2013) – can improve trans- lation, some of the improvement is invisible to an n-gram metric such as BLEU. As a result, tuning to BLEU does not show the full value of a syntactic language model. What does show its value is an op- timization metric that operates on the same syntactic n-grams that are modelled by the dependency LM. The paper is structured as follows. Section 2 de- scribes our relational dependency language model; section 3 describes our neural network training pro- cedure, and the integration of the model into an SMT decoder. We describe the syntactic evaluation metric we use for tuning …

Content truncated to 500 words