Transformer-Based Language Models for Bulgarian

Proceedings of Recent Advances in Natural Language Processing, pages 712–720 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_077 712 Transformer-Based Language Models for Bulgarian Iva Marinova Kiril Simov Institute of Information and Communication Technologies Bulgarian Academy of Sciences Bulgaria iva.marinova@identrics.ai, {kivs|petya}@bultreebank.org Petya Osenova Abstract This paper presents an approach for training lightweight and robust language models for Bulgarian that mitigate gender, political, racial, and other biases in the data. Our method in- volves scraping content from major Bulgar- ian online media providers using a specialized procedure for source filtering, topic selection, and lexicon-based removal of inappropriate lan- guage during the pre-training phase. We con- tinuously improve the models by incorporating new data from various domains, including so- cial media, books, scientific literature, and lin- guistically modified corpora. Our motivation is to provide a solution that is sufficient for all natural language processing tasks in Bulgarian, and to address the lack of existing procedures for guaranteeing the robustness of such models. We evaluated the performance of our language models on several Natural language processing (NLP) tasks, including filling the mask, text generation and named entity recognition (NER). We also performed bias analysis on our mod- els to ensure that they are not biased towards any particular group or ideology. Our analysis showed that within our setting the models have a low level of bias towards gender, race, etc. Needless to say, more experiments have to be performed in future that incorporate compari- son with non-biased data and relies on more bias-related prompts. 1 Introduction Natural language processing has witnessed signifi- cant advancements in recent years, driven by the de- velopment of large-scale pre-trained language mod- els (LMs) such as BERT and GPT-2,3,4. However, such models suffer from biases in the data, which can lead to unfair or discriminatory outputs. Bul- garian language, like many other languages, also lacks robust language models that are not biased to- wards gender, political views, race, or other factors. The rapid advancement in the field, especially in re- cent years, has brought forth unprecedented leaps in the development of high-performance models for various language understanding tasks. However, despite these noteworthy achievements, the NLP re- search community still faces significant challenges, one of which lies in the scarcity of comprehensive and diverse datasets for pre-training Transformer models in less-resourced languages, including Bul- garian. This limitation greatly hinders the other- wise promising potential of these state-of-the-art models to make a profound impact across multiple sectors and geographies. Recognizing the need for a robust and representa- tive dataset for the Bulgarian language is pivotal in addressing this challenge. An ideal dataset should capture the breadth and depth of linguistic diversity, encompassing variations in dialects, registers, and domains. Beyond the level of linguistic parsing, the dataset also needs to reflect the cultural sub- tleties and local phenomena that enrich the texture of the language. Additionally, this dataset must be constructed in a manner that is free from the perils of bias, hate speech, and other problematic elements that would not only undermine the …

Content truncated to 500 words