Cross-Lingual Discriminative Learning of Sequence Models with Posterior Regularization

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1996–2006, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Cross-Lingual Discriminative Learning of Sequence Models with Posterior Regularization Kuzman Ganchev Google Research 76 9th Avenue New York, NY 10011 kuzman@google.com Dipanjan Das Google Research 76 9th Avenue New York, NY 10011 dipanjand@google.com Abstract We present a framework for cross-lingual transfer of sequence information from a resource-rich source language to a resource- impoverished target language that incorporates soft constraints via posterior regularization. To this end, we use automatically word aligned bitext between the source and target language pair, and learn a discriminative conditional ran- dom field model on the target side. Our poste- rior regularization constraints are derived from simple intuitions about the task at hand and from cross-lingual alignment information. We show improvements over strong baselines for two tasks: part-of-speech tagging and named- entity segmentation. 1 Introduction Supervised systems for NLP tasks are available for a handful of languages. These systems achieve high accuracy for many applications; a variety of robust algorithms to train them from labeled data have been developed. Here, we focus on learning sequence mod- els for the languages that lack annotated resources. For a given resource-poor target language of inter- est, we assume that parallel data with a resource-rich source language exists. With the help of this bitext and a supervised system in the source language, we infer constraints over the label distribution in the tar- get language, and train a discriminative model using posterior regularization (Ganchev et al., 2010). Cross-lingual learning of structured prediction models via parallel data has been applied for several natural language processing problems, including part- of-speech (POS) tagging (Yarowsky and Ngai, 2001), syntactic parsing (Hwa et al., 2005) and named-entity recognition (Kim et al., 2012). These methods are useful in several ways. First, they help in fast proto- typing of natural language systems for new languages that do not boast human annotations. Second, the output of such systems could be used to bootstrap more extensive human annotation projects (Vlachos, 2006). Finally, they are significantly more accurate than purely unsupervised systems (McDonald et al., 2011; Das and Petrov, 2011). Recently, T¨ackstr¨om et al. (2013) presented a tech- nique for coupling token constraints derived from pro- jected cross-lingual information and type constraints derived from noisy tag dictionaries to learn POS tag- gers. Although this technique resulted in state-of- the-art weakly supervised taggers, the authors used a heuristic to combine the aforementioned two sources of constraints: the dictionary constraints pruned the tagger’s search space, and the intersected token-level projections were treated as hard observations. On the other hand, Ganchev et al. (2009) presented a framework for learning weakly-supervised systems (in their case, dependency parsers) that incorporated alignment-based information too, but used the cross- lingual information only as soft constraints, via poste- rior regularization. The advantage of this framework lay in the fact that the projections were only trusted to a certain degree, determined by a strength hyper- parameter, …

Content truncated to 500 words