Learning Better Monolingual Models with Unannotated Bilingual Text

Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 46–54, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Learning Better Monolingual Models with Unannotated Bilingual Text David Burkett† Slav Petrov‡ John Blitzer† Dan Klein† †University of California, Berkeley ‡Google Research {dburkett,blitzer,klein}@cs.berkeley.edu slav@google.com Abstract This work shows how to improve state-of-the-art monolingual natural language processing models using unannotated bilingual text. We build a mul- tiview learning objective that enforces agreement between monolingual and bilingual models. In our method the first, monolingual view consists of supervised predictors learned separately for each language. The second, bilingual view consists of log-linear predictors learned over both languages on bilingual text. Our training procedure estimates the parameters of the bilingual model using the output of the monolingual model, and we show how to combine the two models to account for depen- dence between views. For the task of named entity recognition, using bilingual predictors increases F1 by 16.1% absolute over a supervised monolingual model, and retraining on bilingual predictions increases monolingual model F1 by 14.6%. For syntactic parsing, our bilingual predictor increases F1 by 2.1% absolute, and retraining a monolingual model on its output gives an improvement of 2.0%. 1 Introduction Natural language analysis in one language can be improved by exploiting translations in another lan- guage. This observation has formed the basis for important work on syntax projection across lan- guages (Yarowsky et al., 2001; Hwa et al., 2005; Ganchev et al., 2009) and unsupervised syntax induction in multiple languages (Snyder et al., 2009), as well as other tasks, such as cross-lingual named entity recognition (Huang and Vogel, 2002; Moore, 2003) and information retrieval (Si and Callan, 2005). In all of these cases, multilingual models yield increased accuracy because differ- ent languages present different ambiguities and therefore offer complementary constraints on the shared underlying labels. In the present work, we consider a setting where we already possess supervised monolingual mod- els, and wish to improve these models using unan- notated bilingual parallel text (bitext). We cast this problem in the multiple-view (multiview) learning framework (Blum and Mitchell, 1998; Collins and Singer, 1999; Balcan and Blum, 2005; Ganchev et al., 2008). Our two views are a monolingual view, which uses the supervised monolingual models but not bilingual information, and a bilingual view, which exploits features that measure agreement across languages. The parameters of the bilin- gual view are trained to reproduce the output of the monolingual view. We show that by introduc- ing weakened monolingual models into the bilin- gual view, we can optimize the parameters of the bilingual model to improve monolingual models. At prediction time, we automatically account for the between-view dependence introduced by the weakened monolingual models with a simple but effective view-combination heuristic. We demonstrate the performance of this method on two problems. The first is named en- tity recognition (NER). For this problem, our method automatically learns (a variation on) ear- lier hand-designed rule-based bilingual NER pre- dictors (Huang and Vogel, 2002; Moore, 2003), resulting in absolute …

Content truncated to 500 words