From Bag of Languages to Family Trees From Noisy Corpus

International Conference RANLP 2009 - Borovets, Bulgaria, pages 355–359 From Bag of Languages to Family Trees From Noisy Corpus Taraka Rama and Anil Kumar Singh Language Technologies Research Centre, IIIT, Hyderabad, India taraka@students, anil@research@students.iiit.ac.in Abstract In this paper, we use corpus-based measures for constructing phylogenetic trees and try to ad- dress some questions about the validity of do- ing this and applicability to linguistic areas as against language families. We experiment with four corpus based distance measures for con- structing phylogenetic trees. Three of these mea- sures were earlier tried for estimating language distances. We use a fourth measure based on phonetic and orthographic feature n-grams. We compare the trees obtained using these measures and present our observations. Keywords Language distances, similarity measures, phylogenetic trees 1 Introduction Establishing relationships among languages which have been in contact for a long time has been a topic of interest in historical linguistics [6]. However, this topic has been much less explored in the computational lin- guistics community. Most of the previous work is fo- cused on reconstruction of phylogenetic trees for a par- ticular language family using handcrafted word lists [12, 3, 2, 14] or using synthetic data [4]. In this paper we pose the following questions. What happens when we try to construct phylogenetic trees using inter-language distances in the context of a lin- guistic area1? Can the phylogenetic trees be used for evaluating the robustness of the inter-language dis- tance measures and the meaningfulness of the dis- tances? To our knowledge these questions have not been addressed previously. As Singh and Surana [18] showed, corpus based measures can be successfully used for comparative study of languages. Can these distances, estimated from a noisy corpus2, meaning- fully be used to construct phylogenetic trees? Can the information represented by the tree give meaningful interpretations about the languages involved? In this paper, we try to answer these questions. By using meaningful measures for estimating the distance be- tween languages, we try to establish that the answers 1 The term linguistic area or Sprachbund [10] refers to a group of languages that have become similar in some way as a result of proximity and language contact, even if they belong to different families. The best known example is the Indian (or South Asian) linguistic area. 2 By noisy corpus we mean a corpus that includes wrongly spelled words and spelling variations. to these questions are affirmative. Overall, the con- tributions of the paper are the following a) use a new measure for estimating language distance b) present results of the experiments on constructing phyloge- netic trees from corpus based word lists rather than handcrafted ones c) validate the hypothesis that India is a linguistic area [10]. The paper is organized as follows. Related work is discussed in Section 2. A brief discussion of various inter-language measures is given in Section 3. The experimental setup and the analysis of the results have been given in Section 4 and Section 5, respectively. We present a summary of …

Content truncated to 500 words