How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4729–4747 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4729 How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope Yiyun Zhao Department of Linguistics University of Arizona yiyunzhao@arizona.edu Steven Bethard School of Information University of Arizona bethard@arizona.edu Abstract Large pretrained language models like BERT, after fine-tuning to a downstream task, have achieved high performance on a variety of NLP problems. Yet explaining their decisions is difficult despite recent work probing their internal representations. We propose a pro- cedure and analysis methods that take a hy- pothesis of how a transformer-based model might encode a linguistic phenomenon, and test the validity of that hypothesis based on a comparison between knowledge-related down- stream tasks with downstream control tasks, and measurement of cross-dataset consistency. We apply this methodology to test BERT and RoBERTa on a hypothesis that some attention heads will consistently attend from a word in negation scope to the negation cue. We find that after fine-tuning BERT and RoBERTa on a negation scope task, the average attention head improves its sensitivity to negation and its at- tention consistency across negation datasets compared to the pre-trained models. However, only the base models (not the large models) improve compared to a control task, indicat- ing there is evidence for a shallow encoding of negation only in the base models. 1 Introduction As large-scale pre-trained language models such as BERT and ELMo have achieved high performance in a variety of natural language processing tasks (Peters et al., 2018a; Radford et al., 2018; Devlin et al., 2019), a growing body of research is devoted to understanding what linguistic properties these language models have acquired. Recent work uses probes, which are supervised models trained to predict linguistic properties including morphology (Belinkov et al., 2017), syntax (Hewitt and Man- ning, 2019) and semantics (Peters et al., 2018b), etc. (See Belinkov and Glass (2019) for a complete survey.) A good probing performance is considered as evidence that the language models have learned the linguistic knowledge. What is not yet well understood is how this en- coded linguistic knowledge changes when a pre- trained language model is fine-tuned for a down- stream task. Peters et al. (2019) applies a super- vised probe both before and after fine-tuning BERT, and suggests that fine-tuning makes the internal representation task-sensitive. But with supervised probes it can be difficult to disentangle what was learned by the probe from what was present in the internal representation (Hewitt and Liang, 2019). Recent studies have thus turned to unsupervised probes that require no additional training of the model and instead look directly at the attention mechanism, i.e., how much to care about other words when computing the next version of the cur- rent word. Clark et al. (2019) inspected pretrained transformers and found several syntactic properties encoded in an intuitive way, where the maximum attention from a dependent is on its syntactic …

Content truncated to 500 words