Cross-lingual Projected Expectation Regularization for Weakly Supervised Learning

Cross-lingual Projected Expectation Regularization for Weakly Supervised Learning Mengqiu Wang and Christopher D. Manning Computer Science Department Stanford University Stanford, CA 94305 USA {mengqiu,manning}@cs.stanford.edu Abstract We consider a multilingual weakly supervised learning scenario where knowledge from an- notated corpora in a resource-rich language is transferred via bitext to guide the learning in other languages. Past approaches project labels across bitext and use them as features or gold labels for training. We propose a new method that projects model expectations rather than labels, which facilities transfer of model uncertainty across language bound- aries. We encode expectations as constraints and train a discriminative CRF model using Generalized Expectation Criteria (Mann and McCallum, 2010). Evaluated on standard Chinese-English and German-English NER datasets, our method demonstrates F1 scores of 64% and 60% when no labeled data is used. Attaining the same accuracy with su- pervised CRFs requires 12k and 1.5k labeled sentences. Furthermore, when combined with labeled examples, our method yields signifi- cant improvements over state-of-the-art super- vised methods, achieving best reported num- bers to date on Chinese OntoNotes and Ger- man CoNLL-03 datasets. 1 Introduction Supervised statistical learning methods have en- joyed great popularity in Natural Language Process- ing (NLP) over the past decade. The success of su- pervised methods depends heavily upon the avail- ability of large amounts of annotated training data. Manual curation of annotated corpora is a costly and time consuming process. To date, most annotated re- sources resides within the English language, which hinders the adoption of supervised learning methods in many multilingual environments. To minimize the need for annotation, significant progress has been made in developing unsupervised and semi-supervised approaches to NLP (Collins and Singer 1999; Klein 2005; Liang 2005; Smith 2006; Goldberg 2010; inter alia) . More recent paradigms for semi-supervised learning allow mod- elers to directly encode knowledge about the task and the domain as constraints to guide learning (Chang et al., 2007; Mann and McCallum, 2010; Ganchev et al., 2010). However, in a multilingual setting, coming up with effective constraints require extensive knowledge of the foreign1 language. Bilingual parallel text (bitext) lends itself as a medium to transfer knowledge from a resource-rich language to a foreign languages. Yarowsky and Ngai (2001) project labels produced by an English tag- ger to the foreign side of bitext, then use the pro- jected labels to learn a HMM model. More recent work applied the projection-based approach to more language-pairs, and further improved performance through the use of type-level constraints from tag dictionary and feature-rich generative or discrimina- tive models (Das and Petrov, 2011; T¨ackstr¨om et al., 2013). In our work, we propose a new projection-based method that differs in two important ways. First, we never explicitly project the labels. Instead, we project expectations over the labels. This projection 1For experimental purposes, we designate English as the resource-rich language, and other languages of interest as “for- eign”. In our experiments, we simulate the resource-poor sce- nario using Chinese and German, even though in reality these two languages are quite …

Content truncated to 500 words