Unsupervised Structure Prediction with Non-Parallel Multilingual Guidance

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 50–61, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Unsupervised Structure Prediction with Non-Parallel Multilingual Guidance Shay B. Cohen Dipanjan Das Noah A. Smith Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA {scohen,dipanjan,nasmith}@cs.cmu.edu Abstract We describe a method for prediction of lin- guistic structure in a language for which only unlabeled data is available, using annotated data from a set of one or more helper lan- guages. Our approach is based on a model that locally mixes between supervised mod- els from the helper languages. Parallel data is not used, allowing the technique to be ap- plied even in domains where human-translated texts are unavailable. We obtain state-of-the- art performance for two tasks of structure pre- diction: unsupervised part-of-speech tagging and unsupervised dependency parsing. 1 Introduction A major focus of recent NLP research has involved unsupervised learning of structure such as POS tag sequences and parse trees (Klein and Manning, 2004; Johnson et al., 2007; Berg-Kirkpatrick et al., 2010; Cohen and Smith, 2010, inter alia). In its purest form, such research has improved our un- derstanding of unsupervised learning practically and formally, and has led to a wide range of new algo- rithmic ideas. Another strain of research has sought to exploit resources and tools in some languages (es- pecially English) to construct similar resources and tools for other languages, through heuristic “projec- tion” (Yarowsky and Ngai, 2001; Xi and Hwa, 2005) or constraints in learning (Burkett and Klein, 2008; Smith and Eisner, 2009; Das and Petrov, 2011; Mc- Donald et al., 2011) or inference (Smith and Smith, 2004). Joint unsupervised learning (Snyder and Barzilay, 2008; Naseem et al., 2009; Snyder et al., 2009) is yet another research direction that seeks to learn models for many languages at once, exploiting linguistic universals and language similarity. The driving force behind all of this work has been the hope of building NLP tools for languages that lack annotated resources.1 In this paper, we present an approach to using annotated data from one or more languages (helper languages) to learn models for another language that lacks annotated data (the target language). Unlike the previous work mentioned above, our framework does not rely on parallel data in any form. This is advantageous because parallel text exists only in a few text domains (e.g., religious texts, parliamentary proceedings, and news). We focus on generative probabilistic models pa- rameterized by multinomial distributions. We be- gin with supervised maximum likelihood estimates for models of the helper languages. In the second stage, we learn a model for the target language using unannotated data, maximizing likelihood over inter- polations of the helper language models’ distribu- tions. The tying is performed at the parameter level, through coarse, nearly-universal syntactic categories (POS tags). The resulting model is then used to ini- tialize learning of the target language’s model using standard unsupervised parameter estimation. Some previous multilingual research, such as …

Content truncated to 500 words