Minimum Error Rate Training in Statistical Machine Translation

Minimum Error Rate Training in Statistical Machine Translation Franz Josef Och Information Sciences Institute University of Southern California 4676 Admiralty Way, Suite 1001 Marina del Rey, CA 90292 och@isi.edu Abstract Often, the training procedure for statisti- cal machine translation models is based on maximum likelihood or related criteria. A general problem of this approach is that there is only a loose relation to the final translation quality on unseen text. In this paper, we analyze various training criteria which directly optimize translation qual- ity. These training criteria make use of re- cently proposed automatic evaluation met- rics. We describe a new algorithm for effi- cient training an unsmoothed error count. We show that significantly better results can often be obtained if the final evalua- tion criterion is taken directly into account as part of the training procedure. 1 Introduction Many tasks in natural language processing have evaluation criteria that go beyond simply count- ing the number of wrong decisions the system makes. Some often used criteria are, for example, F-Measure for parsing, mean average precision for ranked retrieval, and BLEU or multi-reference word error rate for statistical machine translation. The use of statistical techniques in natural language process- ing often starts out with the simplifying (often im- plicit) assumption that the final scoring is based on simply counting the number of wrong decisions, for instance, the number of sentences incorrectly trans- lated in machine translation. Hence, there is a mis- match between the basic assumptions of the used statistical approach and the final evaluation criterion used to measure success in a task. Ideally, we would like to train our model param- eters such that the end-to-end performance in some application is optimal. In this paper, we investigate methods to efficiently optimize model parameters with respect to machine translation quality as mea- sured by automatic evaluation criteria such as word error rate and BLEU. 2 Statistical Machine Translation with Log-linear Models Let us assume that we are given a source (‘French’) sentence ���� � �� � � � � � � � � , which is to be translated into a target (‘English’) sentence  � � � � � �   � �   Among all possible target sentences, we will choose the sentence with the highest probability:1   �� "!# $ % & Pr ')( �+* (1) The argmax operation denotes the search problem, i.e. the generation of the output sentence in the tar- get language. The decision in Eq. 1 minimizes the number of decision errors. Hence, under a so-called zero-one loss function this decision rule is optimal (Duda and Hart, 1973). Note that using a differ- ent loss function—for example, one induced by the BLEU metric—a different decision rule would be optimal. 1The notational convention will be as follows. We use the symbol Pr ,'- . to denote general probability distributions with (nearly) no specific assumptions. In contrast, for model-based probability distributions, we use the generic symbol /0,'- . . As the true probability …

Content truncated to 500 words