Language Identification: The Long and the Short of the Matter

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 229–237, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Language Identification: The Long and the Short of the Matter Timothy Baldwin and Marco Lui Dept of Computer Science and Software Engineering University of Melbourne, VIC 3010 Australia tb@ldwin.net, saffsd@gmail.com Abstract Language identification is the task of identify- ing the language a given document is written in. This paper describes a detailed examina- tion of what models perform best under dif- ferent conditions, based on experiments across three separate datasets and a range of tokeni- sation strategies. We demonstrate that the task becomes increasingly difficult as we increase the number of languages, reduce the amount of training data and reduce the length of docu- ments. We also show that it is possible to per- form language identification without having to perform explicit character encoding detection. 1 Introduction With the growth of the worldwide web, ever- increasing numbers of documents have become available, in more and more languages. This growth has been a double-edged sword, however, in that content in a given language has become more preva- lent but increasingly hard to find, due to the web’s sheer size and diversity of content. While the ma- jority of (X)HTML documents declare their charac- ter encoding, only a tiny minority specify what lan- guage they are written in, despite support for lan- guage declaration existing in the various (X)HTML standards.1 Additionally, a single encoding can gen- erally be used to render a large number of languages such that the document encoding at best filters out a subset of languages which are incompatible with the given encoding, rather than disambiguates the source language. Given this, the need for automatic means to determine the source language of web doc- 1http://dev.opera.com/articles/view/ mama-head-structure/ uments is crucial for web aggregators of various types. There is widespread misconception of language identification being a “solved task”, generally as a result of isolated experiments over homogeneous datasets with small numbers of languages (Hughes et al., 2006; Xia et al., 2009). Part of the motivation for this paper is to draw attention to the fact that, as a field, we are still a long way off perfect language identification of web documents, as evaluated under realistic conditions. In this paper we describe experiments on lan- guage identification of web documents, focusing on the broad question of what combination of tokenisa- tion strategy and classification model achieves the best overall performance. We additionally evalu- ate the impact of the volume of training data and the test document length on the accuracy of lan- guage identification, and investigate the interaction between character encoding detection and language identification. One assumption we make in this research, follow- ing standard assumptions made in the field, is that all documents are monolingual. This is clearly an un- realistic assumption when dealing with general web documents (Hughes et al., 2006), and we plan to re- turn to investigate language identification over …

Content truncated to 500 words