Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7046–7056, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7046 Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA Ieva Stali¯unait˙e and Ignacio Iacobacci Huawei Noah’s Ark Lab, London, United Kingdom {ieva.staliunaite,ignacio.iacobacci}@huawei.com Abstract Many NLP tasks have benefited from transfer- ring knowledge from contextualized word em- beddings, however the picture of what type of knowledge is transferred is incomplete. This paper studies the types of linguistic phenom- ena accounted for by language models in the context of a Conversational Question Answer- ing (CoQA) task. We identify the problematic areas for the finetuned RoBERTa, BERT and DistilBERT models through systematic error analysis - basic arithmetic (counting phrases), compositional semantics (negation and Seman- tic Role Labeling), and lexical semantics (sur- prisal and antonymy). When enhanced with the relevant linguistic knowledge through mul- titask learning, the models improve in perfor- mance. Ensembles of the enhanced models yield a boost between 2.2 and 2.7 points in F1 score overall, and up to 42.1 points in F1 on the hardest question classes. The results show dif- ferences in ability to represent compositional and lexical information between RoBERTa, BERT and DistilBERT. 1 Introduction It has recently been recognized in the research com- munity that neural network models generally do not exploit the compositionality of language, of- ten relying on superficial features1. Composition- ality refers to the fact that linguistic constituents combine into phrases hierarchically to compose meaning. Contextualized word embeddings (BERT, Devlin et al., 2018; RoBERTa, Liu et al., 2018; Dis- tilBERT, Sanh et al., 2019; etc.) can be expected to be limited in their ability to learn such complex aspects of language, since the models are usually trained with cloze filling and next sentence pre- 1https://2020.ieeeicassp. org/program/plenary-speakers/ deep-representation-learning/ diction objectives, and are not directly exposed to semantic relations between phrases. While larger models yield higher performance, they still lack generalization ability (Talmor et al., 2019) and are computationally expensive, which has led to an increasing interest in reducing model size. For instance, DistilBERT is built using the knowledge distillation technique (Bucil˘a et al., 2006; Hinton et al., 2015) on BERT, which leads to a lighter and faster model that does not lose much in performance on the majority of the tested tasks. On the other hand, state-of-the-art models use vast amounts of training data - 16GB for BERT, 10 times more for RoBERTa. This study tackles the question of what type of linguistic knowledge is missing from contextual- ized word embeddings, comparing models on the basis of their training set size (BERT vs. RoBERTa) as well as their model size (BERT vs. DistilBERT). The tasks of machine reading comprehension (MRC) and dialogue are particularly fitting for this purpose, due to the fact that they require a sys- tem to interpret language within a context and per- form semantic and pragmatic inference between sentences. The task in the Conversational Question Answering dataset (Reddy et al., 2019, CoQA) is …

Content truncated to 500 words