Enhancing Language Models in Statistical Machine Translation with Backward N-grams and Mutual Information Triggers

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 1288–1297, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Enhancing Language Models in Statistical Machine Translation with Backward N-grams and Mutual Information Triggers Deyi Xiong, Min Zhang, Haizhou Li Human Language Technology Institute for Infocomm Research 1 Fusionopolis Way, #21-01 Connexis, Singapore 138632 {dyxiong, mzhang, hli}@i2r.a-star.edu.sg Abstract In this paper, with a belief that a language model that embraces a larger context provides better prediction ability, we present two ex- tensions to standard n-gram language mod- els in statistical machine translation: a back- ward language model that augments the con- ventional forward language model, and a mu- tual information trigger model which captures long-distance dependencies that go beyond the scope of standard n-gram language mod- els. We integrate the two proposed models into phrase-based statistical machine transla- tion and conduct experiments on large-scale training data to investigate their effectiveness. Our experimental results show that both mod- els are able to significantly improve transla- tion quality and collectively achieve up to 1 BLEU point over a competitive baseline. 1 Introduction Language model is one of the most important knowledge sources for statistical machine transla- tion (SMT) (Brown et al., 1993). The standard n-gram language model (Goodman, 2001) assigns probabilities to hypotheses in the target language conditioning on a context history of the preceding n − 1 words. Along with the efforts that advance translation models from word-based paradigm to syntax-based philosophy, in recent years we have also witnessed increasing efforts dedicated to ex- tend standard n-gram language models for SMT. We roughly categorize these efforts into two directions: data-volume-oriented and data-depth-oriented. In the first direction, more data is better. In or- der to benefit from monolingual corpora (LDC news data or news data collected from web pages) that consist of billions or even trillions of English words, huge language models are built in a distributed man- ner (Zhang et al., 2006; Brants et al., 2007). Such language models yield better translation results but at the cost of huge storage and high computation. The second direction digs deeply into monolin- gual data to build linguistically-informed language models. For example, Charniak et al. (2003) present a syntax-based language model for machine transla- tion which is trained on syntactic parse trees. Again, Shen et al. (2008) explore a dependency language model to improve translation quality. To some ex- tent, these syntactically-informed language models are consistent with syntax-based translation models in capturing long-distance dependencies. In this paper, we pursue the second direction with- out resorting to any linguistic resources such as a syntactic parser. With a belief that a language model that embraces a larger context provides better pre- diction ability, we learn additional information from training data to enhance conventional n-gram lan- guage models and extend their ability to capture richer contexts and long-distance dependencies. In particular, we integrate backward n-grams and mu- tual information (MI) triggers into language models in SMT. In conventional n-gram language models, we look …

Content truncated to 500 words