Minimum Imputed-Risk: Unsupervised Discriminative Training for Machine Translation

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 920–929, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Minimum Imputed Risk: Unsupervised Discriminative Training for Machine Translation Zhifei Li∗ Google Research Mountain View, CA 94043, USA zhifei.work@gmail.com Ziyuan Wang, Sanjeev Khudanpur Johns Hopkins University Baltimore, MD 21218, USA zwang40,khudanpur@jhu.edu Jason Eisner Johns Hopkins University Baltimore, MD 21218, USA eisner@jhu.edu Brian Roark Oregon Health & Science University Beaverton, Oregon 97006, USA roark@cslu.ogi.edu Abstract Discriminative training for machine transla- tion has been well studied in the recent past. A limitation of the work to date is that it relies on the availability of high-quality in-domain bilingual text for supervised training. We present an unsupervised discriminative train- ing framework to incorporate the usually plen- tiful target-language monolingual data by us- ing a rough “reverse” translation system. Intu- itively, our method strives to ensure that prob- abilistic “round-trip” translation from a target- language sentence to the source-language and back will have low expected loss. Theoret- ically, this may be justified as (discrimina- tively) minimizing an imputed empirical risk. Empirically, we demonstrate that augment- ing supervised training with unsupervised data improves translation performance over the su- pervised case for both IWSLT and NIST tasks. 1 Introduction Missing data is a common problem in statistics when fitting the parameters θ of a model. A common strat- egy is to attempt to impute, or “fill in,” the missing data (Little and Rubin, 1987), as typified by the EM algorithm. In this paper we develop imputation tech- niques when θ is to be trained discriminatively. We focus on machine translation (MT) as our ex- ample application. A Chinese-to-English machine translation system is given a Chinese sentence x and ∗ Zhifei Li is currently working at Google Research, and this work was done while he was a PHD student at Johns Hop- kins University. asked to predict its English translation y. This sys- tem employs statistical models pθ(y | x) whose pa- rameters θ are discriminatively trained using bilin- gual sentence pairs (x, y). But bilingual data for such supervised training may be relatively scarce for a particular language pair (e.g., Urdu-English), es- pecially for some topics (e.g., technical manuals) or genres (e.g., blogs). So systems seek to exploit ad- ditional monolingual data, i.e., a corpus of English sentences y with no corresponding source-language sentences x, to improve estimation of θ. This is our missing data scenario.1 Discriminative training of the parameters θ of pθ(y | x) using monolingual English data is a cu- rious idea, since there is no Chinese input x to trans- late. We propose an unsupervised training approach, called minimum imputed risk training, which is con- ceptually straightforward: First guess x (probabilis- tically) from the observed y using a reverse English- to-Chinese translation model pφ(x | y). Then train the discriminative Chinese-to-English model pθ(y | x) to do a good job at translating this imputed x back to y, as measured by a given performance met- ric. Intuitively, …

Content truncated to 500 words