Visualizing and Understanding the Effectiveness of BERT

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4143–4152, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4143 Visualizing and Understanding the Effectiveness of BERT Yaru Hao†∗, Li Dong‡, Furu Wei‡, Ke Xu† †Beihang University ‡Microsoft Research {haoyaru@,kexu@nlsde.}buaa.edu.cn {lidong1,fuwei}@microsoft.com Abstract Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks. However, it is unclear why the pre- training-then-fine-tuning paradigm can im- prove performance and generalization capabil- ity across different tasks. In this paper, we pro- pose to visualize loss landscapes and optimiza- tion trajectories of fine-tuning BERT on spe- cific datasets. First, we find that pre-training reaches a good initial point across downstream tasks, which leads to wider optima and eas- ier optimization compared with training from scratch. We also demonstrate that the fine- tuning procedure is robust to overfitting, even though BERT is highly over-parameterized for downstream tasks. Second, the visualization results indicate that fine-tuning BERT tends to generalize better because of the flat and wide optima, and the consistency between the train- ing loss surface and the generalization error surface. Third, the lower layers of BERT are more invariant during fine-tuning, which sug- gests that the layers that are close to input learn more transferable representations of language. 1 Introduction Language model pre-training has achieved strong performance in many NLP tasks (Peters et al., 2018; Howard and Ruder, 2018a; Radford et al., 2018; Devlin et al., 2018; Baevski et al., 2019; Dong et al., 2019). A neural encoder is trained on a large text corpus by using language model- ing objectives. Then the pre-trained model either is used to extract vector representations for input, or is fine-tuned on the specific datasets. Recent work (Tenney et al., 2019b; Liu et al., 2019a; Goldberg, 2019; Tenney et al., 2019a) has shown that the pre-trained models can en- code syntactic and semantic information of lan- guage. However, it is unclear why pre-training ∗Contribution during internship at Microsoft Research. is effective on downstream tasks in terms of both trainability and generalization capability. In this work, we take BERT (Devlin et al., 2018) as an example to understand the effectiveness of pre- training. We visualize the loss landscapes and the optimization procedure of fine-tuning on spe- cific datasets in three ways. First, we compute the one-dimensional (1D) loss curve, so that we can inspect the difference between fine-tuning BERT and training from scratch. Second, we visualize the two-dimensional (2D) loss surface, which pro- vides more information about loss landscapes than 1D curves. Third, we project the high-dimensional optimization trajectory of fine-tuning to the ob- tained 2D loss surface, which demonstrate the learning properties in an intuitive way. The main findings are summarized as follows. First, visualization results indicate that BERT pre- training reaches a good initial point across down- stream tasks, which leads to wider optima on the 2D loss landscape compared with random initial- ization. Moreover, the visualization of optimiza- …

Content truncated to 500 words