Language ID in the Context of Harvesting Language Data off the Web

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 870–878, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Language ID in the Context of Harvesting Language Data off the Web Fei Xia University of Washington Seattle, WA 98195, USA fxia@u.washington.edu William D. Lewis Microsoft Research Redmond, WA 98052, USA wilewis@microsoft.com Hoifung Poon University of Washington Seattle, WA 98195, USA hoifung@cs.washington.edu Abstract As the arm of NLP technologies extends beyond a small core of languages, tech- niques for working with instances of lan- guage data across hundreds to thousands of languages may require revisiting and re- calibrating the tried and true methods that are used. Of the NLP techniques that has been treated as “solved” is language iden- tification (language ID) of written text. However, we argue that language ID is far from solved when one considers in- put spanning not dozens of languages, but rather hundreds to thousands, a number that one approaches when harvesting lan- guage data found on the Web. We formu- late language ID as a coreference resolu- tion problem and apply it to a Web harvest- ing task for a specific linguistic data type and achieve a much higher accuracy than long accepted language ID approaches. 1 Introduction A large number of the world’s languages have been documented by linguists; it is now increas- ingly common to post current research and data to the Web, often in the form of language snip- pets embedded in scholarly papers. A particu- larly common format for linguistic data posted to the Web is “interlinearized text”, a format used to present language data and analysis relevant to a particular argument or investigation. Since in- terlinear examples consist of orthographically or phonetically encoded language data aligned with an English translation, the “corpus” of interlinear examples found on the Web, when taken together, constitute a significant multilingual, parallel cor- pus covering hundreds to thousands of the world’s languages. Previous work has discussed methods for harvesting interlinear text off the Web (Lewis, 2006), enriching it via structural projections (Xia and Lewis, 2007), and even making it available to typological analyses (Lewis and Xia, 2008) and search (Xia and Lewis, 2008). One challenge with harvesting interlinear data off the Web is language identification of the har- vested data. There have been extensive studies on language identification (language ID) of writ- ten text, and a review of previous research on this topic can be found in (Hughes et al., 2006). In gen- eral, a language ID method requires a collection of text for training, something on the order of a thousand or more characters. These methods work well for languages with rich language resources; for instance, Cavnar and Trenkle’s N-gram-based algorithm achieved an accuracy as high as 99.8% when tested on newsgroup articles across eight languages (Cavnar and Trenkle, 1994). However, the performance is much worse (with accuracy dropping to as low as 1.66%) if there is very lit- tle language data for training and the …

Content truncated to 500 words