Nested Named Entity Recognition via Second-best Sequence Learning and Decoding

What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models Allyson Ettinger Department of Linguistics University of Chicago aettinger@uchicago.edu Abstract Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about information used by language models for generating pre- dictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared cate- gory or role reversal, albeit with less sensitiv- ity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inference and role-based event prediction— and, in particular, it shows clear insensitivity to the contextual impacts of negation. 1 Introduction Pre-training of NLP models with a language mod- eling objective has recently gained popularity as a precursor to task-specific fine-tuning. Pre- trained models like BERT (Devlin et al., 2019) and ELMo (Peters et al., 2018a) have advanced the state of the art in a wide variety of tasks, suggesting that these models acquire valuable, generalizable linguistic competence during the pre-training process. However, though we have established the benefits of language model pre- training, we have yet to understand what exactly about language these models learn during that process. This paper aims to improve our understanding of what language models (LMs) know about lan- guage, by introducing a set of diagnostics target- ing a range of linguistic capacities drawn from human psycholinguistic experiments. Because of their origin in psycholinguistics, these diagnostics have two distinct advantages: They are carefully controlled to ask targeted questions about linguis- tic capabilities, and they are designed to ask these questions by examining word predictions in con- text, which allows us to study LMs without any need for task-specific fine-tuning. Beyond these advantages, our diagnostics dis- tinguish themselves from existing tests for LMs in two primary ways. First, these tests have been chosen specifically for their capacity to reveal insensitivities in predictive models, as evidenced by patterns that they elicit in human brain re- sponses. Second, each of these tests targets a set of linguistic capacities that extend beyond the primarily syntactic focus seen in existing LM diagnostics—we have tests targeting com- monsense/pragmatic inference, semantic roles and event knowledge, category membership, and ne- gation. Each of our diagnostics is set up to sup- port tests of both word prediction accuracy and sensitivity to distinctions between good and bad context completions. Although we focus on the BERT model here as an illustrative case study, these diagnostics are applicable for testing of any language model. This paper makes two main contributions. First, we introduce a new set of targeted diagnostics for assessing linguistic capacities in language models.1 Second, we apply these tests to shed light on strengths and weaknesses of the popular BERT …

Content truncated to 500 words