Language identification of names with SVMs

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 693–696, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Language identification of names with SVMs Aditya Bhargava and Grzegorz Kondrak Department of Computing Science University of Alberta Edmonton, Alberta, Canada, T6G 2E8 {abhargava,kondrak}@cs.ualberta.ca Abstract The task of identifying the language of text or utterances has a number of applications in natural language processing. Language iden- tification has traditionally been approached with character-level language models. How- ever, the language model approach crucially depends on the length of the text in ques- tion. In this paper, we consider the problem of language identification of names. We show that an approach based on SVMs with n-gram counts as features performs much better than language models. We also experiment with applying the method to pre-process transliter- ation data for the training of separate models. 1 Introduction The task of identifying the language of text or utter- ances has a number of applications in natural lan- guage processing. Font Llitj´os and Black (2001) show that language identification can improve the accuracy of letter-to-phoneme conversion. Li et al. (2007) use language identification in a translit- eration system to account for different semantic transliteration rules between languages when the tar- get language is Chinese. Huang (2005) improves the accuracy of machine transliteration by clustering his training data according to the source language. Language identification has traditionally been approached using character-level n-gram language models. In this paper, we propose the use of sup- port vector machines (SVMs) for the language iden- tification of very short texts such as proper nouns. We show that SVMs outperform language models on two different data sets consisting of personal names. Furthermore, we test the hypothesis that lan- guage identification can improve transliteration by pre-processing the source data and training separate models using a state-of-the-art transliteration sys- tem. 2 Previous work N-gram approaches have proven very popular for language identification in general. Cavnar and Tren- kle (1994) apply n-gram language models to general text categorization. They construct character-level language models using n-grams up to a certain max- imum length from each class in their training cor- pora. To classify new text, they generate an n-gram frequency profile from the text and then assign it to the class having the most similar language model, which is determined by summing the differences in n-gram ranks. Given 14 languages, text of 300 char- acters or more, and retaining the 400 most common n-grams up to length 5, they achieve an overall accu- racy of 99.8%. However, the accuracy of the n-gram approach strongly depends on the length of the texts. Kruengkrai et al. (2005) report that, on a language identification task of 17 languages with average text length 50 bytes, the accuracy drops to 90.2%. When SVMs were used for the same task, they achieved 99.7% accuracy. Konstantopoulos (2007) looks particularly at the task of identifying the language of proper nouns. He focuses on a data set of …

Content truncated to 500 words