Language Identification of Search Engine Queries

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 1066–1074, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Language Identification of Search Engine Queries Hakan Ceylan Department of Computer Science University of North Texas Denton, TX, 76203 hakan@unt.edu Yookyung Kim Yahoo! Inc. 2821 Mission College Blvd. Santa Clara, CA, 95054 ykim@yahoo-inc.com Abstract We consider the language identification problem for search engine queries. First, we propose a method to automatically generate a data set, which uses click- through logs of the Yahoo! Search En- gine to derive the language of a query indi- rectly from the language of the documents clicked by the users. Next, we use this data set to train two decision tree classi- fiers; one that only uses linguistic features and is aimed for textual language identi- fication, and one that additionally uses a non-linguistic feature, and is geared to- wards the identification of the language intended by the users of the search en- gine. Our results show that our method produces a highly reliable data set very ef- ficiently, and our decision tree classifier outperforms some of the best methods that have been proposed for the task of written language identification on the domain of search engine queries. 1 Introduction The language identification problem refers to the task of deciding in which natural language a given text is written. Although the problem is heav- ily studied by the Natural Language Processing community, most of the research carried out to date has been concerned with relatively long texts such as articles or web pages which usually con- tain enough text for the systems built for this task to reach almost perfect accuracy. Figure 1 shows the performance of 6 different language identifi- cation methods on written texts of 10 European languages that use the Roman Alphabet. It can be seen that the methods reach a very high ac- curacy when the text has 100 or more characters. However, search engine queries are very short in length; they have about 2 to 3 words on average, Figure 1: Performance of six Language Identifica- tion methods on varying text size. Adapted from (Poutsma, 2001). which requires a reconsideration of the existing methods built for this problem. Correct identification of the language of the queries is of critical importance to search engines. Major search engines such as Yahoo! Search (www.yahoo.com), or Google (www.google.com) crawl billions of web pages in more than 50 lan- guages, and about a quarter of their queries are in languages other than English. Therefore a correct identification of the language of a query is needed in order to aid the search engine towards more ac- curate results. Moreover, it also helps further pro- cessing of the queries, such as stemming or spell checking of the query terms. One of the challenges in this problem is the lack of any standard or publicly available data set. Fur- thermore, creating such a data set is expensive as it requires an …

Content truncated to 500 words