Clustered language models based on regular expressions for SMT

Clustered Language Models based on Regular Expressions for SMT Saˇsa Hasan and Hermann Ney Lehrstuhl f¨ur Informatik VI – Computer Science Department RWTH Aachen University 52056 Aachen, Germany {hasan,ney}@cs.rwth-aachen.de Abstract. In this paper, we present a language model based on clusters obtained by applying regular expressions to the training data and, thus, discriminating several different sentence types as, e.g. interrog- atives, imperatives or enumerations. The main motivation lies in the observation that different sentence types also underlie a different syntactic structure, and thus yield a varying distribution of n-grams reflect- ing their word order. We show that this assumption is valid by applying the models to English-Spanish bilingual corpora and obtaining good perplexity reductions of approximately 25%. In addition, we per- form an n-best rescoring experiment and show a relative improvement of 4-5% in word error rate. The models can be easily adapted to other translation tasks and do not need complicated training methods, thus being a valuable alternative for on-demand rescoring of sentence hypotheses such as they occur in the CAT framework. 1 Introduction Language modeling is a rather long-established re- search field in the area of Natural Language Pro- cessing. In Automatic Speech Recognition (ASR), the language model guides the acoustic analysis by specifying the order in which a sequence of words is likely to occur. In Statistical Machine Transla- tion (SMT), where the best translation ˆeI 1 of source words fJ 1 is obtained by maximizing the condi- tional probability ˆeI 1 = argmax eI 1 {Pr(eI 1|fJ 1 )} = argmax eI 1 {Pr(fJ 1 |eI 1) · Pr(eI 1)} (1) by using Bayes decision rule, the first probability on the right-hand side of the equation denotes the translation model whereas the second is the lan- guage model of the target language. The prevalent approach of modeling Pr(eI 1) is based on n-grams with suitable smoothing methods: Pr(eI 1) = I� i=1 Pr(ei|ei−1 i−n+1) (2) Due to the problem of sparse data, high order n- grams, i.e. where n > 3, are rarely applied. Thus, only local syntactic dependencies are captured in a conventional trigram or bigram. In the past, additional models have been pro- posed that boost the performance of simple tri- gram models. It is shown in (Martin et al., 1999; Goodman, 2000) that a combination of individual techniques based on caching, skipping, clustering or sentence mixtures improves the baseline signifi- cantly. In this approach, we will present a clustering technique that is based on regular expressions. The motivation behind this lies in the following obser- vation: the syntactic structure of a sentence is influ- enced by its type. It is obvious that an interrogative sentence has a different structure from a declarative one due to non-local dependencies arising e.g. from wh-extraction. As an example, consider the syntax of the following sentences: • What are distribution templates? • Distribution templates are what were previ- ously referred to as templates or scan tem- plates. If we look closer at the first four words of each …

Content truncated to 500 words