CxGBERT: BERT meets Construction Grammar

Proceedings of the 28th International Conference on Computational Linguistics, pages 4020–4032 Barcelona, Spain (Online), December 8-13, 2020 4020 CxGBERT: BERT meets Construction Grammar Harish Tayyar Madabushi1 and Laurence Romain2 and Dagmar Divjak2,3 and Petar Milin2 1 School of Computer Science 2 Department of Modern Languages 3 Department of English Language and Linguistics University of Birmingham, UK Harish@HarishTayyarMadabushi.com (L.M.Y.Romain, D.Divjak, P.Milin)@bham.ac.uk Abstract While lexico-semantic elements no doubt capture a large amount of linguistic information, it has been argued that they do not capture all information contained in text. This assumption is central to constructionist approaches to language which argue that language consists of constructions, learned pairings of a form and a function or meaning that are either frequent or have a meaning that cannot be predicted from its component parts. BERT’s training objectives give it access to a tremendous amount of lexico-semantic information, and while BERTology has shown that BERT captures certain important linguistic dimensions, there have been no studies exploring the extent to which BERT might have access to constructional information. In this work we design several probes and conduct extensive experiments to answer this question. Our results allow us to conclude that BERT does indeed have access to a significant amount of information, much of which linguists typically call constructional information. The impact of this observation is potentially far-reaching as it provides insights into what deep learning methods learn from text, while also showing that information contained in constructions is redundantly encoded in lexico- semantics. 1 Introduction and Motivation The introduction of pre-trained contextual word embeddings has had a tremendous impact on Natural Language Processing (NLP), resulting in significant improvements on several tasks such as translation and question answering, where the best automated systems are at par with or outperform humans (Hassan et al., 2018). These models can broadly be classified based on the approach they use: the pre-training approach and the feature-based approach. While methods that rely on features, such as ELMo (Peters et al., 2018) require task specific architectures, those that rely on fine-tuning, such as Generative Pre- trained Transformer (OpenAI GPT) (Radford et al., 2018), update all parameters during fine-tuning thus transferring their learning to new tasks with minimal fine-tuning. While not the first pre-trained model, BERT (Devlin et al., 2018) was the first extremely successful one, utilising a deep bidirectional transformer model to provide significant improvements on the state-of-the-art of several tasks. The success of BERT and its effectiveness in transfer learning, which is inherent to pre-trained models, could imply that these models have an understanding of the underlying language. This has led several researchers to focus on explaining what it is that these models understand about language. Despite the inherent difficulty in understanding deep learning models, a new subfield of NLP, called BERTol- ogy (Rogers et al., 2020), has evolved to better understand what BERT captures within its structure and how it is able to so effectively transfer that knowledge to so many different tasks through relatively quick fine-tuning. These efforts have shown that …

Content truncated to 500 words