Cross-domain Feature Selection for Language Identification

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 553–561, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Cross-domain Feature Selection for Language Identification Marco Lui and Timothy Baldwin NICTA VRL Department of Computer Science and Software Engineering University of Melbourne, VIC 3010, Australia saffsd@gmail.com, tb@ldwin.net Abstract We show that transductive (cross-domain) learning is an important consideration in building a general-purpose language iden- tification system, and develop a feature selection method that generalizes across domains. Our results demonstrate that our method provides improvements in transductive transfer learning for language identification. We provide an implementa- tion of the method and show that our sys- tem is faster than popular standalone lan- guage identification systems, while main- taining competitive accuracy. 1 Introduction Language identification (LangID) is the task of determining the language(s) that a text is written in. It is considered by some researchers to be a solved task, because previous research has re- ported near-perfect accuracy (Cavnar and Trenkle, 1994). Hughes et al. (2006) elaborated a number of simplifying assumptions that have made this the case, and Baldwin and Lui (2010a) showed that when some of these assumptions are relaxed to make the task closer to the actuality of open-web LangID, it becomes considerably harder. In this paper, we demonstrate that the style of evaluation used by Baldwin and Lui (2010a) performs well in-domain but badly cross-domain, and develop a novel method for preserving high in-domain ac- curacy, while significantly boosting cross-domain accuracy. Similarly to Baldwin and Lui (2010a), we make the simplifying assumption that all doc- uments are monolingual, despite recent work on multilingual LangID (Baldwin and Lui, 2010b). LangID is usually formulated as a supervised machine learning problem and evaluated in an of- fline setting (Cavnar and Trenkle, 1994; Baldwin and Lui, 2010a). However, its primary use is online without any additional configuration, op- timized for maximal cross-domain accuracy. number of such standalone LangID systems areA available, notable among which is TextCat (van Noord, 1997). TextCat has been the LangID solution of choice in research, and is the basis of language identification/filtering in the ClueWeb09 Dataset (Callan and Hoy, 2009) and Corpus- Builder (Ghani et al., 2004). Elsewhere, Google provides LangID as a web service via its Google Language Detect API (GoogleAPI). While it has much higher accuracy than TextCat (as we show in Section 6.1), research applications contravene the service’s terms of use, and moreover the ser- vice is rate-limited. Our ideal system should offer the same ad- vantages as TextCat in terms of licensing and run-time speed, while matching the open-domain accuracy and ease-of-use of an API such as GoogleAPI. To this end, we release the op- timized final LangID system described in this paper, and benchmark it against TextCat and GoogleAPI. Our major contributions are: (1) we show that negative transfer occurs when training a classifier over a combined set of LangID datasets; (2) we show that language models learned in a par- ticular domain do not always generalize to other …

Content truncated to 500 words