Applying Natural Annotation and Curriculum Learning to Named Entity Recognition for Under-Resourced Languages

Proceedings of the 29th International Conference on Computational Linguistics, pages 4468–4480 October 12–17, 2022. 4468 Applying Natural Annotation and Curriculum Learning to Named Entity Recognition for Under-Resourced Languages Valeriy Lobov MIPT lobov.va@phystech.edu Alexandra Ivoylova RSUH a.m.ivoylova@gmail.com Serge Sharoff School of Languages University of Leeds, UK s.sharoff@leeds.ac.uk Abstract Current practices in building new NLP mod- els for low-resourced languages rely either on Machine Translation of training sets from bet- ter resourced languages or on cross-lingual transfer from them. Still we can see a con- siderable performance gap between the mod- els originally trained within better resourced languages and the models transferred from them. In this study we test the possibility of (1) using natural annotation to build syn- thetic training sets from resources not initially designed for the target downstream task and (2) employing curriculum learning methods to select the most suitable examples from syn- thetic training sets. We test this hypothesis across seven Slavic languages and across three curriculum learning strategies on Named En- tity Recognition as the downstream task. We also test the possibility of fine-tuning the syn- thetic resources to reflect linguistic proper- ties, such as the grammatical case and gender, both of which are important for the Slavic languages. We demonstrate the possibility to achieve the mean F1 score of 0.78 across the three basic entities types for Belarus- ian starting from zero resources in compar- ison to the baseline of 0.63 using the zero- shot transfer from English. For comparison, the English model trained on the original set achieves the mean F1-score of 0.75. The experimental results are available from ht- tps://github.com/ValeraLobov/SlavNER 1 Introduction The use of pre-trained language models (PLMs), such as BERT (Devlin et al., 2018), has signi- ficantly improved accuracy of many NLP tasks, such as POS tagging and Named Entity Recog- nition (Tenney et al., 2019). It is also possible to achieve good quality transfer of the classifi- ers across the languages using multilingual PLMs (Conneau et al., 2020; Hu et al., 2020). How- ever, lesser resourced languages still present a con- siderable problem. First, the amount of raw text data available for their pretraining is on the order of magnitudes smaller than what is available for bigger languages, such as English. Second, zero- or few-shot multilingual transfer comes with the price of a performance gap, when a model tested on lesser resourced recipient languages is less ac- curate than the original donor language model. This comes partly because of the linguistic differ- ences between the donor and recipient languages and partly because of the lower quality of pre- trained embeddings obtained on smaller corpora for lesser resourced recipient languages (Vuli´c et al., 2020). We propose a method to improve the accuracy of models for lesser resourced languages by what we call “natural" annotation, i.e., when some de- sired linguistic properties are derived from annota- tions arising as a by-product of a natural activ- ity which is not directly related to the task of the model. Parallel corpora provide an example of …

Content truncated to 500 words