Language comparison through sparse multilingual word alignment

Proceedings of the EACL 2012 Joint Workshop of LINGVIS & UNCLH, pages 54–62, Avignon, France, April 23 - 24 2012. c⃝2012 Association for Computational Linguistics Language comparison through sparse multilingual word alignment Thomas Mayer Research Unit Quantitative Language Comparison LMU Munich thommy.mayer@googlemail.com Michael Cysouw Research Center Deutscher Sprachatlas Philipp University of Marburg cysouw@uni-marburg.de Abstract In this paper, we propose a novel approach to compare languages on the basis of par- allel texts. Instead of using word lists or abstract grammatical characteristics to infer (phylogenetic) relationships, we use mul- tilingual alignments of words in sentences to establish measures of language similar- ity. To this end, we introduce a new method to quickly infer a multilingual alignment of words, using the co-occurrence of words in a massively parallel text (MPT) to simulta- neously align a large number of languages. The idea is that a simultaneous multilin- gual alignment yields a more adequate clus- tering of words across different languages than the successive analysis of bilingual alignments. Since the method is computa- tionally demanding for a larger number of languages, we reformulate the problem us- ing sparse matrix calculations. The useful- ness of the approach is tested on an MPT that has been extracted from pamphlets of the Jehova’s Witnesses. Our preliminary experiments show that this approach can supplement both the historical and the ty- pological comparison of languages. 1 Introduction The application of quantitative methods in histor- ical linguistics has attracted a lot of attention in recent years (cf. Steiner et al. (2011) for a sur- vey). Many ideas have been adapted from evolu- tionary biology and bioinformatics, where similar problems occur with respect to the genealogical grouping of species and the multiple alignment of strings/sequences. One of the main differences between those areas and attempts to uncover lan- guage history is the limited amount of suitable data that can serve as the basis for language com- parison. A widely used resource are Swadesh lists or similar collections of translational equivalents in the form of word lists. Likewise, phylogenetic methods have been applied using structural char- acteristics (e.g., Dunn et al. (2005)). In this paper, we propose yet another data source, namely par- allel texts. Many analogies have been drawn between the evolution of species and languages (see, for in- stance, Pagel (2009) for such a comparison). One of the central problems is to establish what is the equivalent of the gene in the reproduction of lan- guages. Like in evolutionary biology, where gene sequences in organisms are compared to infer phylogenetic trees, a comparison of the “genes” of language would be most appropriate for a quan- titative analysis of languages. Yet, Swadesh- like wordlists or structural characteristics do not neatly fit into this scheme as they are most likely not the basis on which languages are replicated. After all, language is passed on as the expression of propositions, i.e. sentences, which usually con- sists of more than single words. Hence, follow- ing Croft (2000), we assume that the basic unit of …

Content truncated to 500 words