Attention Can Reflect Syntactic Structure (If You Let It)

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3031–3045 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3031 Attention Can Reflect Syntactic Structure (If You Let It) ∗Vinit Ravishankar† ∗Artur Kulmizev‡ Mostafa Abdou§ Anders Søgaard§ Joakim Nivre‡ † Language Technology Group, Department of Informatics, University of Oslo ‡ Department of Linguistics and Philology, Uppsala University § Department of Computer Science, University of Copenhagen † vinitr@ifi.uio.no ‡ {artur.kulmizev,joakim.nivre}@lingfil.uu.se § {abdou,soegaard}@di.ku.dk Abstract Since the popularization of the Transformer as a general-purpose feature encoder for NLP, many studies have attempted to decode lin- guistic structure from its novel multi-head at- tention mechanism. However, much of such work focused almost exclusively on English — a language with rigid word order and a lack of inflectional morphology. In this study, we present decoding experiments for multilingual BERT across 18 languages in order to test the generalizability of the claim that dependency syntax is reflected in attention patterns. We show that full trees can be decoded above base- line accuracy from single attention heads, and that individual relations are often tracked by the same heads across languages. Furthermore, in an attempt to address recent debates about the status of attention as an explanatory mecha- nism, we experiment with fine-tuning mBERT on a supervised parsing objective while freez- ing different series of parameters. Interest- ingly, in steering the objective to learn explicit linguistic structure, we find much of the same structure represented in the resulting attention patterns, with interesting differences with re- spect to which parameters are frozen. 1 Introduction In recent years, the attention mechanism proposed by Bahdanau et al. (2014) has become an indis- pensable component of many NLP systems. Its widespread adoption was, in part, heralded by the introduction of the Transformer architecture (Vaswani et al., 2017a), which constrains a soft alignment to be learned across discrete states in the input (self-attention), rather than across input and output (e.g., Xu et al., 2015; Rockt¨aschel et al., 2015). The Transformer has, by now, supplanted the popular LSTM (Hochreiter and Schmidhuber, ∗Equal contribution. Order was decided by a coin toss. 1997) as NLP’s feature-encoder-of-choice, largely due to its compatibility with parallelized training regimes and ability to handle long-distance depen- dencies. Certainly, the nature of attention as a distribu- tion over tokens lends itself to a straightforward interpretation of a model’s inner workings. Bah- danau et al. (2014) illustrate this nicely in the con- text of seq2seq machine translation, showing that the attention learned by their models reflects expected cross-lingual idiosyncrasies between En- glish and French, e.g., concerning word order. With self-attentive Transformers, interpretation becomes slightly more difficult, as attention is distributed across words within the input itself. This is fur- ther compounded by the use of multiple layers and heads, each combination of which yields its own alignment, representing a different (possibly re- dundant) view of the data. Given the similarity of such attention matrices to the score matrices employed in arc-factored dependency parsing (Mc- Donald …

Content truncated to 500 words