On the evolution of syntactic information encoded by BERT’s contextualized representations

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2243–2258 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2243 On the Evolution of Syntactic Information Encoded by BERT’s Contextualized Representations Laura P´erez-Mayos1, Roberto Carlini1, Miguel Ballesteros2, Leo Wanner3,1 1 TALN Research Group, Pompeu Fabra University, Barcelona, Spain 2 Amazon AI 3 Catalan Institute for Research and Advanced Studies (ICREA), Barcelona, Spain {laura.perezm|roberto.carlini|leo.wanner}@upf.edu ballemig@amazon.com Abstract The adaptation of pretrained language models to solve supervised tasks has become a base- line in NLP, and many recent works have fo- cused on studying how linguistic information is encoded in the pretrained sentence repre- sentations. Among other information, it has been shown that entire syntax trees are implic- itly embedded in the geometry of such mod- els. As these models are often fine-tuned, it becomes increasingly important to understand how the encoded knowledge evolves along the fine-tuning. In this paper, we analyze the evolution of the embedded syntax trees along the fine-tuning process of BERT for six dif- ferent tasks, covering all levels of the linguis- tic structure. Experimental results show that the encoded syntactic information is forgot- ten (PoS tagging), reinforced (dependency and constituency parsing) or preserved (semantics- related tasks) in different ways along the fine- tuning process depending on the task. 1 Introduction Adapting unsupervised pretrained language mod- els (LMs) to solve supervised tasks has become a widely spread practice in NLP, with models such as ELMo (Peters et al., 2018) and, most notably, BERT (Devlin et al., 2019), achieving state-of-the-art results in many well-known Natural Language Understanding benchmarks like GLUE (Wang et al., 2018) and SQuAD (Rajpurkar et al., 2018). Several studies investigate what the LMs learn, how and where the learned knowledge is represented and what the best methods to improve it are; cf., e.g., (Rogers et al., 2020). There is evidence that, among other information (such as, e.g., PoS, syntactic chunks and roles (Tenney et al., 2019b; Lin et al., 2019; Belinkov et al., 2017), mor- phology in general (Peters et al., 2018), or sentence length (Adi et al., 2016)) BERT deep models’ vec- tor geometry implicitly embeds entire syntax trees (Hewitt and Manning, 2019). However, rather lit- tle is understood about how these representations change when fine-tuned to solve downstream tasks (Peters et al., 2019). In this work, we aim to understand how syn- tax trees implicitly embedded in the geometry of deep models evolve along the fine-tuning process of BERT on different supervised tasks, and shed some light on the importance of the syntactic in- formation for those tasks. Intuitively, we expect morpho-syntactic tasks to clearly reinforce the en- coded syntactic information, while tasks that are not explicitly syntactic in nature should maintain it in case they benefit from syntax (Kuncoro et al., 2020) and lose it if they do not. In order to cover the three main levels of the linguistic description (morphology, syntax and semantics), we select six different tasks: PoS tagging, constituency pars- ing, syntactic …

Content truncated to 500 words