Behavior of Modern Pre-trained Language Models Using the Example of Probing Tasks

Proceedings of Recent Advances in Natural Language Processing, pages 664–670 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_075 664 Abstract Modern transformer-based language models are revolutionizing NLP. However, existing studies into language modelling with BERT have been mostly limited to English-language material and do not pay enough attention to the implicit knowledge of language, such as semantic roles, presupposition and negations, that can be acquired by the model during training. Thus, the aim of this study is to examine behavior of the model BERT in the task of masked language modelling and to provide linguistic interpretation to the unexpected effects and errors produced by the model. For this purpose, we used a new Russian- language dataset based on educational texts for learners of Russian and annotated with the help of the National Corpus of the Russian language. In terms of quality metrics (the proportion of words, semantically related to the target word), the multilingual BERT is recognized as the best model. Generally, each model has distinct strengths in relation to a certain linguistic phenomenon. These observations have meaningful implications for research into applied linguistics and pedagogy, contribute to dialogue system development, automatic exercise making, text generation and potentially could improve the quality of existing linguistic technologies. 1 Introduction As is well-known, 2018 saw a breakthrough in natural language processing (NLP) with the advent of several novel pre-trained language 1 All datasets and the Jupyter notebook with the experiments are made available as a GitLab repository: https://gitlab.com/lieutkat/linguistic-experiments-with-bert/ models, including BERT (Devlin et al, 2018). These models are capable of fine-tuning, that is, additional training on a smaller dataset for a specific task. Such models develop a certain degree of understanding natural language and therefore require further studies (Zhang et al, 2019; Wallat et al, 2020). To this end, the so- called probing tasks are commonly used, such as testing the model’s ability to identify semantically coherent sentences in running text. A considerable amount of recent research (Gulordava et al, 2018; Salazar et al, 2019, Sun et al, 2019) was devoted to the general language modelling problem and also to masked language modelling. Other scholars (Devlin, 2018; Gong, 2019; Rogers et al, 2020) explored BERT specifically. However, there is a lack of studies concerning BERT’s behavior on Russian language material. In addition, existing papers discuss the problem without taking into consideration systematic linguistic features. This paper aims to explore the behavior of the pre-trained BERT language model using the example of the diagnostic task of masked language modeling for the Russian language and give a linguistic interpretation of the cases when the model shows unsatisfactory results. This study will focus on new language data with experiment design based on linguistic theories1. The problem will be discussed in terms of the language modelling concept, cognitive science, theory of language, in particular the language frames theory, semantic roles concept, and also negations processing depending on a context. 2 Related Work Bengio and co-authors (Bengio et al, 2003) suggest the following definition of a language Behavior of Modern Pre-trained …

Content truncated to 500 words