IgboBERT Models: Building and Training Transformer Models for the Igbo Language

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5114–5122 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5114 IgboBERT Models: Building and Training Transformer Models for the Igbo Language Chiamaka Chukwuneke1,2, Ignatius Ezeani1,2, Paul Rayson1 and Mahmoud El-Haj1 1UCREL NLP Group, Lancaster University 2Dept. of Computer Sc. Nnamdi Azikiwe University, Anambra State, Nigeria {c.chukwuneke, i.ezeani, p.rayson, m.el-haj}@lancaster.ac.uk Abstract This work presents a standard Igbo named entity recognition (IgboNER) dataset as well as the results from training and fine-tuning state-of-the-art transformer IgboNER models. We discuss the process of our dataset creation - data collection and annotation and quality checking. We also present experimental processes involved in building an IgboBERT language model from scratch as well as fine-tuning it along with other non-Igbo pre-trained models for the downstream IgboNER task. Our results show that, although the IgboNER task benefited hugely from fine-tuning large transformer model, fine-tuning a transformer model built from scratch with comparatively little Igbo text data seems to yield quite decent results for the IgboNER task. This work will contribute immensely to IgboNLP in particular as well as the wider African and low-resource NLP efforts. Keywords: Igbo, named entity recognition, BERT models, under-resourced, dataset 1. Introduction The African continent has over 2,000 languages (Eber- hard et al., 2020) and information is stored digitally in many of those languages. To digitally interact in those languages, localisation of computer interfaces and tools are very vital and this leads to the need for Natural Lan- guage Processing (NLP) research to build these tools. The African continent is underrepresented in the NLP research (Adelani et al., 2021) despite these numer- ous languages spoken in its 54 countries. Some of the contributing factors to lack of research in these coun- tries include very few available language resources and computers with computing capacity to handle such re- search. This limits the development and creation of tools and resources for performing a wide variety of NLP tasks such as named entity recognition (NER), machine translation (MT), information retrieval etc. This paper focuses on building resources - data set and models - for IgboNER i.e. named entity recognition for Igbo, a language mainly spoken in the south east- ern part of Nigeria. Named Entity Recognition is a term defined as the task of identifying names of or- ganizations, people, currency, time, percentage expres- sion and geographic locations in text which was intro- duced at the sixth Message Understanding Conference (MUC-6) (Grishman and Sundheim, 1995). NER is a very important pre-processing step in key NLP tasks such as Question Answering (Mollá et al., 2006), In- formation Retrieval (Guo et al., 2009), Automatic Text Summarization (Nobata et al., 2002), Machine Trans- lation (Babych and Hartley, 2003) etc. This work is an extension of the on-going efforts (Onyenwe et al., 2014; Ezeani et al., 2016; Ezeani et al., 2020; Onyenwe et al., 2018), and we present the Igbo part of the MasakhaNER dataset to support Igbo NLP in particular but also …

Content truncated to 500 words