Automatic Detection and Language Identification of Multilingual Documents

Automatic Detection and Language Identification of Multilingual Documents Marco Lui♥♣, Jey Han Lau♠ and Timothy Baldwin♥♣ ♥ Department of Computing and Information Systems The University of Melbourne ♣ NICTA Victoria Research Laboratory ♠ Department of Philosophy King’s College London mhlui@unimelb.edu.au, jeyhan.lau@gmail.com, tb@ldwin.net Abstract Language identification is the task of automat- ically detecting the language(s) present in a document based on the content of the docu- ment. In this work, we address the problem of detecting documents that contain text from more than one language (multilingual docu- ments). We introduce a method that is able to detect that a document is multilingual, iden- tify the languages present, and estimate their relative proportions. We demonstrate the ef- fectiveness of our method over synthetic data, as well as real-world multilingual documents collected from the web. 1 Introduction Language identification is the task of automatically detecting the language(s) present in a document based on the content of the document. Language identification techniques commonly assume that ev- ery document is written in one of a closed set of known languages for which there is training data, and is thus formulated as the task of selecting the most likely language from the set of training lan- guages. In this work, we remove this monolingual assumption, and address the problem of language identification in documents that may contain text from more than one language from the candidate set. We propose a method that concurrently detects that a document is multilingual, and estimates the propor- tion of the document that is written in each language. Detecting multilingual documents has a variety of applications. Most natural language processing techniques presuppose monolingual input data, so inclusion of data in foreign languages introduces noise, and can degrade the performance of NLP sys- tems (Alex et al., 2007; Cook and Lui, 2012). Au- tomatic detection of multilingual documents can be used as a pre-filtering step to improve the quality of input data. Detecting multilingual documents is also important for acquiring linguistic data from the web (Scannell, 2007; Abney and Bird, 2010), and has applications in mining bilingual texts for statistical machine translation from online resources (Resnik, 1999; Nie et al., 1999; Ling et al., 2013). There has been particular interest in extracting text resources for low-density languages from multilingual web pages containing both the low-density language and another language such as English (Yamaguchi and Tanaka-Ishii, 2012; King and Abney, 2013). King and Abney (2013, p1118) specifically mention the need for an automatic method “to examine a mul- tilingual document, and with high accuracy, list the languages that are present in the document”. We introduce a method that is able to detect multi- lingual documents, and simultaneously identify each language present as well as estimate the propor- tion of the document written in that language. We achieve this with a probabilistic mixture model, us- ing a document representation developed for mono- lingual language identification (Lui and Baldwin, 2011). The model posits that each document is gen- erated as samples from an unknown mixture …

Content truncated to 500 words