Adding More Languages Improves Unsupervised Multilingual Part-of-Speech Tagging: a Bayesian Non-Parametric Approach

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 83–91, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Adding More Languages Improves Unsupervised Multilingual Part-of-Speech Tagging: A Bayesian Non-Parametric Approach Benjamin Snyder, Tahira Naseem, Jacob Eisenstein, and Regina Barzilay Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology {bsnyder, tahira, jacobe, regina}@csail.mit.edu Abstract We investigate the problem of unsupervised part-of-speech tagging when raw parallel data is available in a large number of languages. Patterns of ambiguity vary greatly across lan- guages and therefore even unannotated multi- lingual data can serve as a learning signal. We propose a non-parametric Bayesian model that connects related tagging decisions across lan- guages through the use of multilingual latent variables. Our experiments show that perfor- mance improves steadily as the number of lan- guages increases. 1 Introduction In this paper we investigate the problem of unsu- pervised part-of-speech tagging when unannotated parallel data is available in a large number of lan- guages. Our goal is to develop a fully joint multilin- gual model that scales well and shows improved per- formance for individual languages as the total num- ber of languages increases. Languages exhibit ambiguity at multiple levels, making unsupervised induction of their underlying structure a difficult task. However, sources of lin- guistic ambiguity vary across languages. For exam- ple, the word fish in English can be used as either a verb or a noun. In French, however, the noun pois- son (fish) is entirely distinct from the verbal form pˆecher (to fish). Previous work has leveraged this idea by building models for unsupervised learning from aligned bilingual data (Snyder et al., 2008). However, aligned data is often available for many languages. The benefits of bilingual learning vary markedly depending on which pair of languages is selected, and without labeled data it is unclear how to determine which supplementary language is most helpful. In this paper, we show that it is possi- ble to leverage all aligned languages simultaneously, achieving accuracy that in most cases outperforms even optimally chosen bilingual pairings. Even in expressing the same meaning, languages take different syntactic routes, leading to variation in part-of-speech sequences. Therefore, an effec- tive multilingual model must accurately model com- mon linguistic structure, yet remain flexible to the idiosyncrasies of each language. This tension only becomes stronger as additional languages are added to the mix. From a computational standpoint, the main challenge is to ensure that the model scales well as the number of languages increases. Care must be taken to avoid an exponential increase in the parameter space as well as the time complexity of inference procedure. We propose a non-parametric Bayesian model for joint multilingual tagging. The topology of our model connects tagging decisions within a language as well as across languages. The model scales lin- early with the number of languages, allowing us to incorporate as many as are available. For each lan- guage, the model contains an HMM-like substruc- ture and connects these substructures …

Content truncated to 500 words