Multilingual Language Identification: ALTW 2010 Shared Task Data

Multilingual Language Identification: ALTW 2010 Shared Task Dataset Timothy Baldwin and Marco Lui NICTA VRL Department of Computer Science and Software Engineering University of Melbourne, VIC 3010, Australia tb@ldwin.net, saffsd@gmail.com Abstract While there has traditionally been strong interest in the task of monolingual lan- guage identification, research on multi- lingual language identification is under- represented in the literature, partly due to a lack of standardised datasets. This paper describes an artificially-generated dataset for multilingual language identification, as used in the 2010 Australasian Language Technology Workshop shared task. 1 Introduction Language identification is traditionally defined as the task of determining the unique language a given document is authored in, under the assump- tion that all documents are monolingual (Bald- win and Lui, to appear). In contexts such as the web, however, multilingual documents are com- monplace, suggesting the need for language iden- tification research to move towards a more realis- tic task setting where a document can be authored in one or more languages (Hughes et al., 2006). This paper describes such a dataset, based around the task of multilingual language identification, where the task is to determine which one or two languages a given document is authored in. This dataset formed the basis of the 2010 Australasian Language Technology Workshop shared task. Multilingual language identification is relevant in a number of contexts. “Word spotting” of for- eign words in multilingual documents has been shown to improve parsing performance (Alex et al., 2007), and multilingual language identifica- tion is a first step in this direction. It can also be used as part of a linguistic corpus creation pipeline for low-density languages, e.g. to determine the language used in interlinear glossed text (IGT) embedded in language documentation (Xia et al., 2009; Xia and Lewis, 2009). The ideal vehicle for multilingual language identification research would be a dataset gen- uinely representative of the true multilingualism of resources such as the web. Creating such a resource, however, would require: (a) a multilin- gual crawl without language bias; and (b) a large- scale document collection with gold-standard an- notations over the full range of languages extant on the web, including sub-document extents for the individual languages contained in a document. While we would ultimately like to generate such a dataset for general usage, in this paper we de- scribe a more modest effort to artificially gener- ate a dataset for multilingual language identifica- tion purposes. Our basic approach is to: (1) select a language bias-preserving set of primary docu- ments; (2) select a comparable document for each in a second language based on translation links; and (3) concatenate sections of the two documents together to form a single multilingual document. In this paper, we detail the methodology for gener- ating the dataset, and outline baseline and bench- mark results over the dataset to calibrate future ef- forts. 2 Dataset Synthesis The dataset for the task was prepared from database exports of the various language Wikipedias provided by the WikiMedia Founda- tion.1 The WikiMedia Foundation carries …

Content truncated to 500 words