Language Identification for Person Names Based on Statistical Information

Proceedings of PACLIC 19, the 19th Asia-Pacific Conference on Language, Information and Computation. Language Identification for Person Names Based on Statistical Information Shiho Nobesawa Department of Information Sciences Tokyo University of Science 2641 Yamazaki, Noda Chiba, 278-8510, Japan shiho@is.noda.tus.ac.jp Ikuo Tahara Department of Information Sciences Tokyo University of Science 2641 Yamazaki, Noda Chiba, 278-8510, Japan Tahara@is.noda.tus.ac.jp Abstract Language identification has been an interesting and fascinating issue in natural language processing for decades, and there have been many researches on it. However, most of the researches are for documents, and though the possibility of high accuracy for shorter strings of characters, language identification for words or phrases has not been discussed much. In this paper we propose a statistical method of language identification for phrases, and show the empirical results for person names of 9 languages (12 areas). Our simple method based on n-gram and phrase length obtained more than 90% of accuracy for Japanese, Korean and Russian, and fair results for other languages except English. This result indicated the possibility of language identification for person names based on statistics, which is useful in multi-language person name detection and also let us expect the possibility of language identification for phrases with simple statistics-based methods. 1. Introduction The technology of language identification has become more important with the growth of the WWW. As Grefenstette reported in their paper (Grefenstette 2000) non-English languages are growing in recent years on the WWW, and the need for automatic language identification for both documents and phrases are increasing. An easy method for language identification must be a key for better accuracy rate in natural language processing, such as information retrieval and machine translation. Language identification is not a new topic in natural language processing. Bastrup proposed a unigram-based decision-tree method for language identification (Bastrup 2003). Dunning reported that 20 bytes are enough to obtain 92% accuracy in language identification (Dunning 1994). His method was based on statistical information, and it did not use any accented characters which would be a great help in identifying. This result is very encouraging for applying statistical language identification to proper nouns. Language identification is also well examined as speech recognition task (Matrouf 1998, Schultz 1996, Hazen 1994a, Hazen 1994b, Lamel 1994, Berkling 1994). Caseiro and Trancoso introduced a method using one language phone recognizer and less linguistic information for the language identification of speech (Caseiro 1998). 2. Language Identification for Person Names Person name is one of the most frequent foreign phrases in texts, and it often causes noise as unknown words. Thus language identification of person names can be a help for better accuracy in text processing. However, there are several difficulties in identifying the language of a person name. First, the language of a person name may not match the official language of the area. Second, person names are international and not as language-specific as other words. Third, person names are often not long enough for analysis, and it is even possible to have name words in different …

Content truncated to 500 words