Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus

Proceedings of the 28th International Conference on Computational Linguistics, pages 6588–6608 Barcelona, Spain (Online), December 8-13, 2020 6588 Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus Isaac Caswell, Theresa Breiner, Daan van Esch, Ankur Bapna Google Research, 1600 Amphitheatre Parkway, Mountain View, CA 94043 {icaswell,tbreiner,dvanesch,ankurbpn}@google.com Abstract Large text corpora are increasingly important for a wide variety of Natural Language Process- ing (NLP) tasks, and automatic language identification (LangID) is a core technology needed to collect such datasets in a multilingual context. LangID is largely treated as solved in the literature, with models reported that achieve over 90% average F1 on as many as 1,366 lan- guages. We train LangID models on up to 1,629 languages with comparable quality on held-out test sets, but find that human-judged LangID accuracy for web-crawl text corpora created using these models is only around 5% for many lower-resource languages, suggesting a need for more robust evaluation. Further analysis revealed a variety of error modes, arising from domain mis- match, class imbalance, language similarity, and insufficiently expressive models. We propose two classes of techniques to mitigate these errors: wordlist-based tunable-precision filters (for which we release curated lists in about 500 languages) and transformer-based semi-supervised LangID models, which increase median dataset precision from 5.5% to 71.2%. These techniques enable us to create an initial data set covering 100K or more relatively clean sentences in each of 500+ languages, paving the way towards a 1,000-language web text corpus. 1 Introduction Thousands of languages are spoken in our world (Eberhard et al., 2019), but technologies like machine translation (MT) and automatic speech recognition (ASR) are only available in about 100 of them. As internet access becomes increasingly common with the spread of smartphones (Biggs, 2017), bringing technologies that can help lower language and literacy barriers to more languages is ever more important. Unfortunately, bringing language technologies to more languages is costly, as for many technologies, extending to an additional language has generally required the use of large parallel labeled datasets. For example, ASR systems are usually trained on large sets of audio recordings and transcriptions, while MT systems have historically needed a set of bilingual sentence pairs. Increasingly, small parallel datasets do exist for many languages (Mayer and Cysouw, 2014; Agi´c and Vuli´c, 2019; Artetxe et al., 2020; Ardila et al., 2020), but those resources were either produced at high cost, or are restricted to narrow domains. Parallel resources, which rarely occur naturally, remain scarce for most languages. Monolingual text data, which is more commonly produced, is also used in building out language technologies: for example, in training language models, which are used in many applications ranging from next-word prediction in keyboard input software (Ouyang et al., 2017) to ASR and MT (Buck et al., 2014). Historically, though, a monolingual text corpus by itself has not been sufficient to build ASR and MT systems in a new language: at least some parallel data was typically necessary. Recently, however, significant progress has …

Content truncated to 500 words