Language Identification for Austronesian Languages

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6530–6539 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6530 Language Identification for Austronesian Languages Jonathan Dunn, Wikke Nijhof University of Canterbury, Department of Linguistics and the New Zealand Institute for Language, Brain and Behaviour Christchurch, New Zealand jonathan.dunn@canterbury.ac.nz, wni20@uclive.ac.nz Abstract This paper provides language identification models for low- and under-resourced languages in the Pacific region with a focus on previously unavailable Austronesian languages. Accurate language identification is an important part of developing language resources. The approach taken in this paper combines 29 Austronesian languages with 171 non-Austronesian languages to create an evaluation set drawn from eight data sources. After evaluating six approaches to language identification, we find that a classifier based on skip-gram embeddings reaches a significantly higher performance than alternate methods. We then systematically increase the number of non-Austronesian languages in the model up to a total of 800 languages to evaluate whether an increased language inventory leads to less precise predictions for the Austronesian languages of interest. This evaluation finds that there is only a minimal impact on accuracy caused by increasing the inventory of non-Austronesian languages. Further experiments adapt these language identification models for code-switching detection, achieving high accuracy across all 29 languages. Keywords: language identification, Austronesian languages, code-switching detection, low-resource languages 1. Identifying Pacific Languages Language identification (LID) remains an important problem within natural language processing because it is a central component in the creation of many cor- pora. The result is that languages which are not covered by a LID model also lack the data and resources neces- sary for many applications. This lack of data cannot be solved by bootstrapping methods (Baroni et al., 2009; Goldhahn et al., 2012; Benko, 2014) because it is not possible to identify new samples of the language. This means that accurate LID for minority languages has been and continues to be an important challenge (Jauhiainen et al., 2019), often the first step in developing resources for low-resource languages. This paper addresses the problem of language iden- tification for the Pacific region, with a special focus on low- and under-resourced Polynesian and Austrone- sian languages. We provide a LID model capable of accurately identifying 29 Polynesian/Austronesian lan- guages against a selection of languages likely to occur in the Pacific region (200 in total). Of these Austronesian languages, 9 have not been previously available in LID models. An additional challenge is that the Austrone- sian languages of interest are closely related: similar, for example, to the challenge of identifying related Uralic languages (Chakravarthi et al., 2021). This makes the identification task more difficult because the languages themselves are relatively similar. Even for Austronesian languages that have been included in previous models, then, this work improves our ability to distinguish them from a wider range of closely related languages. The experiments in this paper first evaluate six LID archi- tectures on an inventory of 200 languages to see which approaches work best …

Content truncated to 500 words