Universal Dependencies According to BERT: Both More Specific and More General

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2710–2722 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2710 Universal Dependencies According to BERT: Both More Specific and More General Tomasz Limisiewicz and David Mareˇcek and Rudolf Rosa Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics Charles University, Prague, Czech Republic {limisiewicz,rosa,marecek}@ufal.mff.cuni.cz Abstract This work focuses on analyzing the form and extent of syntactic abstraction captured by BERT by extracting labeled dependency trees from self-attentions. Previous work showed that individual BERT heads tend to encode particular dependency relation types. We extend these findings by explicitly comparing BERT relations to Uni- versal Dependencies (UD) annotations, show- ing that they often do not match one-to-one. We suggest a method for relation identification and syntactic tree construction. Our approach produces significantly more consistent depen- dency trees than previous work, showing that it better explains the syntactic abstractions in BERT. At the same time, it can be successfully ap- plied with only a minimal amount of supervi- sion and generalizes well across languages. 1 Introduction and Related Work In recent years, systems based on Transformer ar- chitecture achieved state-of-the-art results in lan- guage modeling (Devlin et al., 2019) and machine translation (Vaswani et al., 2017). Additionally, the contextual embeddings obtained from the interme- diate representation of the model brought improve- ments in various NLP tasks. Multiple recent works try to analyze such latent representations (Linzen et al., 2019), observe syntactic properties in some Transformer self-attention heads, and extract syn- tactic trees from the attentions matrices (Raganato and Tiedemann, 2018; Mareˇcek and Rosa, 2019; Clark et al., 2019; Jawahar et al., 2019). In our work, we focus on the comparative anal- ysis of the syntactic structure, examining how the BERT self-attention weights correspond to Uni- versal Dependencies (UD) syntax (Nivre et al., 2016). We confirm the findings of Vig and Be- linkov (2019) and Voita et al. (2019) that in Trans- former based systems particular heads tend to cap- ture specific dependency relation types (e.g. in one head the attention at the predicate is usually focused on the nominal subject). We extend understanding of syntax in BERT by examining the ways in which it systematically di- verges from standard annotation (UD). We attempt to bridge the gap between them in three ways: • We modify the UD annotation of three lin- guistic phenomena to better match the BERT syntax (§3) • We introduce a head ensemble method, com- bining multiple heads which capture the same dependency relation label (§4) • We observe and analyze multipurpose heads, containing multiple syntactic functions (§7) Finally, we apply our observations to improve the method of extracting dependency trees from attention (§5), and analyze the results both in a monolingual and a multilingual setting (§6). Our method crucially differs from probing (Be- linkov et al., 2017; Hewitt and Manning, 2019; Chi et al., 2020; Kulmizev et al., 2020). We do not use treebank data to train a parser; rather, we extract dependency relations directly …

Content truncated to 500 words