GeezSwitch: Language Identification in Typologically Related Low-resourced East African Languages

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6578–6584 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6578 GeezSwitch: Language Identification in Typologically Related Low-resourced East African Languages Fitsum Gaim, Wonsuk Yang, Jong C. Park Korea Advanced Institute of Science and Technology Daejeon, South Korea {fgaim, dirrick0511, park}@nlp.kaist.ac.kr Abstract Language identification is one of the fundamental tasks in natural language processing that is a prerequisite to data processing and numerous applications. Low-resourced languages with similar typologies are generally confused with each other in real-world applications such as machine translation, affecting the user’s experience. In this work, we present a language- identification dataset for five typologically and phylogenetically related low-resourced East African languages that use the Ge’ez script as a writing system; namely Amharic, Blin, Ge’ez, Tigre, and Tigrinya. The dataset is built automatically from selected data sources, but we also performed a manual evaluation to assess its quality. Our approach to constructing the dataset is cost-effective and applicable to other low-resource languages. We integrated the dataset into an existing language-identification tool and also fine-tuned several Transformer based language models, achieving very strong results in all cases. While the task of language identification is easy for the informed person, such datasets can make a difference in real-world deployments and also serve as part of a benchmark for language understanding in the target languages. The data and models are made available at https://github.com/fgaim/geezswitch. Keywords: language-identification, low-resource, multilingual models, Amharic, Blin, Ge’ez, Tigre, Tigrinya 1. Introduction Language Identification (LI) is the task of determining the language in which a document or snippet of text is written; and it is a well-established task in the fields of Natural Language Processing (NLP) and Information Retrieval (IR) (Gold, 1967). Generally, text processing techniques presuppose that the language of the input text is known and that all the inputs are in the same lan- guage. However, in order to apply these techniques to real-world data, automatic LI is needed to ensure that only content in the relevant languages is subjected to further processing. Recently, NLP applications have seen tremendous progress thanks to the availability of large amounts of relevant data and methods. However, fundamental tasks such as LI are often overlooked for low-resourced languages, particularly for those with non-Latin writing systems. Applications such as ma- chine translation that do not have proper LI capability in place tend to make embarrassing mistakes that neg- atively affect the user’s experience. This is a common experience for the native speakers of languages such as Tigrinya and Amharic, where the languages are often confused with each other by online services. The Ge’ez script is an abugida writing system where each letter represents a consonant-vowel syllable, re- ferred to locally as Fidel. The script was originally used to write the Ge‘ez language, which is now con- sidered extinct as a vernacular language and circum- scribed to liturgical purposes. However, the Ge’ez lan- guage is survived by several languages of Eritrea and Ethiopia …

Content truncated to 500 words