HeLI-OTS, Off-the-shelf Language Identifier for Text

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3912–3922 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3912 HeLI-OTS, Off-the-shelf Language Identifier for Text Tommi Jauhiainen, Heidi Jauhiainen, Krister Lind´en University of Helsinki {firstname.lastname}@helsinki.fi Abstract This paper introduces HeLI-OTS, an off-the-shelf text language identification tool using the HeLI language identification method. The HeLI-OTS language identifier is equipped with language models for 200 languages and licensed for academic as well as commercial use. We present the HeLI method and its use in our previous research. Then we compare the performance of the HeLI-OTS language identifier with that of fastText on two different data sets, showing that fastText favors the recall of common languages, whereas HeLI-OTS reaches both high recall and high precision for all languages. While introducing existing off-the-shelf language identification tools, we also give a picture of digital humanities-related research that uses such tools. The validity of the results of such research depends on the results given by the language identifier used, and especially for research focusing on the less common languages, the tendency to favor widely used languages might be very detrimental, which Heli-OTS is now able to remedy. Keywords: language identification, text classification 1. Introduction The significance of language identification as part of text preprocessing has grown at the same time as natu- ral language processing (NLP) systems have become more common in widely used software applications. For example, to perform machine translation on a piece of text, the machine translation system must know the source text’s language. Without automated language identification, the users have to indicate the language of the text manually. Google Translate1 is an exam- ple of a system where language identification has been incorporated (Jauhiainen, 2019). Language identification is a many-faceted task, and the specific aspect that the system presented in this paper sets out to tackle is the most general case of reliably identifying a large group of languages, most of which are not closely related. The HeLI-OTS, an off-the-shelf language identifier based on the HeLI method (Jauhi- ainen et al., 2016), includes a repertoire of 200 lan- guages. In Section 2, we introduce language identification as well as some of the existing off-the-shelf language identification tools. Section 3 presents the HeLI method together with some of the results of the pre- vious language identification performance evaluations where the technique has been featured. In Section 4, we describe the HeLI-OTS tool itself, and in Section 5, we evaluate the performance of the current implementation and compare it with that of the fastText language identi- fier (Joulin et al., 2016). Section 6 brings to light some known limitations of the tool, Section 7 concludes our findings and Section 8 lists our current improvement targets for future versions of HeLI-OTS. 1https://translate.google.com/ 2. Previous Work 2.1. Language Identification The problem of identifying the language of digital text has been researched since the 1960s (Jauhiainen et al., 2019d). It can be considered a sub-task of text classi- …

Content truncated to 500 words