Probing for Labeled Dependency Trees

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7711 - 7726 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Probing for Labeled Dependency Trees Max Müller-Eberstein and Rob van der Goot and Barbara Plank Department of Computer Science IT University of Copenhagen, Denmark mamy@itu.dk, robv@itu.dk, bapl@itu.dk Abstract Probing has become an important tool for an- alyzing representations in Natural Language Processing (NLP). For graphical NLP tasks such as dependency parsing, linear probes are currently limited to extracting undirected or unlabeled parse trees which do not capture the full task. This work introduces DEPPROBE, a linear probe which can extract labeled and directed dependency parse trees from embed- dings while using fewer parameters and com- pute than prior methods. Leveraging its full task coverage and lightweight parametrization, we investigate its predictive power for select- ing the best transfer language for training a full biaffine attention parser. Across 13 languages, our proposed method identifies the best source treebank 94% of the time, outperforming com- petitive baselines and prior work. Finally, we analyze the informativeness of task-specific subspaces in contextual embeddings as well as which benefits a full parser’s non-linear parametrization provides. 1 Introduction Pre-trained, contextualized embeddings have been found to encapsulate information relevant to var- ious syntactic and semantic tasks out-of-the-box (Tenney et al., 2019; Hewitt and Manning, 2019). Quantifying this latent information has become the task of probes — models which take frozen em- beddings as input and are parametrized as lightly as possible (e.g. linear transformations). Recent proposals for edge probing (Tenney et al., 2019) and structural probing (Hewitt and Manning, 2019) have enabled analyses beyond classification tasks, including graphical tasks such as dependency pars- ing. They are able to extract dependency graphs from embeddings, however these are either undi- rected (Hewitt and Manning, 2019; Hall Maudslay et al., 2020) or unlabeled (Kulmizev et al., 2020), thereby capturing only a subset of the full task. root punct obj nsubj amod det root nsubj det amod obj punct DEPPROBE Embeddings B L Figure 1: DEPPROBE extracts tree structure using transformation B, labels using L and infers directional- ity using root, based on contextualized embeddings. In this work, we investigate whether this gap can be filled and ask: Can we construct a lightweight probe which can produce fully directed and labeled dependency trees? Using these trees, we further aim to study the less examined problem of trans- ferability estimation for graphical tasks, extend- ing recent work targeting classification and regres- sion tasks (Nguyen et al., 2020; You et al., 2021). Specifically: How well do our probe’s predictions correlate with the transfer performance of a full parser across a diverse set of languages? To answer these questions, we contribute DEP- PROBE (Figure 1), the first linear probe to extract directed and labeled dependency trees while using fewer parameters than prior work and three orders of magnitude fewer trainable parameters than a full parser (Section 3). As this allows us to measure labeled attachment scores …

Content truncated to 500 words