LanideNN: Multilingual Language Identification on Character Window

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 927–936, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics LanideNN: Multilingual Language Identification on Character Window Tom Kocmi and Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {kocmi,bojar}@ufal.mff.cuni.cz Abstract In language identification, a common first step in natural language processing, we want to automatically determine the lan- guage of some input text. Monolingual language identification assumes that the given document is written in one language. In multilingual language identification, the document is usually in two or three lan- guages and we just want their names. We aim one step further and propose a method for textual language identification where languages can change arbitrarily and the goal is to identify the spans of each of the languages. Our method is based on Bidirectional Re- current Neural Networks and it performs well in monolingual and multilingual lan- guage identification tasks on six datasets covering 131 languages. The method keeps the accuracy also for short docu- ments and across domains, so it is ideal for off-the-shelf use without preparation of training data. 1 Introduction The World Wide Web is an ever growing source of textual data, especially data generated by web users. As more people get access to the web, more languages and dialects start to appear and need to be processed. In order to be able to use such data for further natural language processing (NLP) tasks, we need to know in which languages they were written. Language identification is thus a key component for both building various NLP resources from the web and also for running many web services. Techniques of language identification can rely on handcrafted rules, usually of high precision but low coverage, or data-driven methods that learn to identify languages based on sample texts of suffi- cient quantity. In this paper, we present a data-driven method for language identification based on bidirectional recurrent neural networks called LanideNN (lan- guage identification by neural networks, NN). The model is trained on character sliding window of input texts with the goal of assigning a language to each character. We show that the method is applicable for a large number of languages and across text domains without any adaptation and that it performs well in monolingual (one lan- guage per document) as well as multilingual (a few languages per document) language identification tasks. Also, the performance does not drop with shorter texts. The paper is structured as follows. In Section 2, we briefly review current approaches to language identification. Section 3 introduces our method, including the technical details of the neural net- work architecture. For the training of our model, we collect and manually clean a new dataset, as described in Section 4. The model is evaluated on standard test sets for monolingual (Section 5) as well as multilingual (Section 6) language identi- fication. Section 7 illustrates the behavior of our method in …

Content truncated to 500 words