Generative Models of Monolingual and Bilingual Gappy Patterns

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 512–522, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Generative Models of Monolingual and Bilingual Gappy Patterns Kevin Gimpel Noah A. Smith Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213, USA {kgimpel,nasmith}@cs.cmu.edu Abstract A growing body of machine translation re- search aims to exploit lexical patterns (e.g., n- grams and phrase pairs) with gaps (Simard et al., 2005; Chiang, 2005; Xiong et al., 2011). Typically, these “gappy patterns” are discov- ered using heuristics based on word align- ments or local statistics such as mutual infor- mation. In this paper, we develop generative models of monolingual and parallel text that build sentences using gappy patterns of arbi- trary length and with arbitrarily many gaps. We exploit Bayesian nonparametrics and col- lapsed Gibbs sampling to discover salient pat- terns in a corpus. We evaluate the patterns qualitatively and also add them as features to an MT system, reporting promising prelimi- nary results. 1 Introduction Beginning with the success of phrase-based transla- tion models (Koehn et al., 2003), a trend arose of modeling larger and increasingly complex structural units in translation. One thread of work has focused on the use of lexical patterns with gaps. Simard et al. (2005) proposed using phrase pairs with gaps in a phrase-based translation model, providing a heuris- tic method to extract gappy phrase pairs from word- aligned parallel corpora. The widely-used hierarchi- cal phrase-based translation framework was intro- duced by Chiang (2005) and also relies on a simple heuristic for phrase pair extraction. On the mono- lingual side, researchers have taken inspiration from trigger-based language modeling for speech recog- nition (Rosenfeld, 1996). Recently Xiong et al. (2011) used monolingual trigger pairs to improve handling of long-distance dependencies in machine translation output. All of this previous work used heuristics or local statistical tests to extract patterns from corpora. In this paper, we present probabilistic models that gen- erate text using gappy patterns of arbitrary length and with arbitrarily-many gaps. We exploit non- parametric priors and use Bayesian inference to dis- cover the most salient gappy patterns in monolin- gual and parallel text. We first inspect these pat- terns manually and discuss the categories of phe- nomena that they capture. We also add them as features in a discriminatively-trained phrase-based MT system, using standard techniques to train their weights (Arun and Koehn, 2007; Watanabe et al., 2007) and incorporate them during decoding (Chi- ang, 2007). We present experiments for Spanish- English and Chinese-English translation, reporting encouraging preliminary results. 2 Related Work There is a rich history of trigger-based language modeling in the speech recognition community, typ- ically involving the use of statistical tests to discover useful trigger-word pairs (Rosenfeld, 1996; Jelinek, 1997). Xiong et al. (2011) used Rosenfeld’s mutual information procedure to discover trigger pairs and added a single feature to a phrase-based MT system that scores new words based on all potential trig- gers from previous parts of the derivation. We are not aware …

Content truncated to 500 words