Do Syntax Trees Help Pre-trained Transformers Extract Information?

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2647–2661 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2647 Do Syntax Trees Help Pre-trained Transformers Extract Information? Devendra Singh Sachan1,2, Yuhao Zhang3, Peng Qi3, William Hamilton1,2 1Mila - Quebec AI Institute 2School of Computer Science, McGill University 3Stanford University sachande@mila.quebec, wlh@cs.mcgill.ca {yuhaozhang, pengqi}@stanford.edu Abstract Much recent work suggests that incorporat- ing syntax information from dependency trees can improve task-specific transformer models. However, the effect of incorporating depen- dency tree information into pre-trained trans- former models (e.g., BERT) remains unclear, especially given recent studies highlighting how these models implicitly encode syntax. In this work, we systematically study the util- ity of incorporating dependency trees into pre- trained transformers on three representative in- formation extraction tasks: semantic role label- ing (SRL), named entity recognition, and rela- tion extraction. We propose and investigate two distinct strate- gies for incorporating dependency structure: a late fusion approach, which applies a graph neural network on the output of a transformer, and a joint fusion approach, which infuses syn- tax structure into the transformer attention lay- ers. These strategies are representative of prior work, but we introduce additional model de- sign elements that are necessary for obtaining improved performance. Our empirical anal- ysis demonstrates that these syntax-infused transformers obtain state-of-the-art results on SRL and relation extraction tasks. However, our analysis also reveals a critical shortcom- ing of these models: we find that their perfor- mance gains are highly contingent on the avail- ability of human-annotated dependency parses, which raises important questions regarding the viability of syntax-augmented transformers in real-world applications.1 1 Introduction Dependency trees—a form of syntactic represen- tation that encodes an asymmetric syntactic rela- tion between words in a sentence, such as sub- 1Our code is available at: https://github.com/ DevSinghSachan/syntax-augmented-bert ject or adverbial modifier—have proven very use- ful in various NLP tasks. For instance, features defined in terms of the shortest path between entities in a dependency tree were used in rela- tion extraction (RE) (Fundel et al., 2006; Björne et al., 2009), parse structure has improved named entity recognition (NER) (Jie et al., 2017), and joint parsing was shown to benefit semantic role labeling (SRL) (Pradhan et al., 2005) systems. More recently, dependency trees have also led to meaningful performance improvements when in- corporated into neural network models for these tasks. Popular encoders to include dependency tree into neural models include graph neural networks (GNNs) for SRL (Marcheggiani and Titov, 2017) and RE (Zhang et al., 2018), and biaffine attention in transformers for SRL (Strubell et al., 2018). In parallel, there has been a renewed interest in investigating self-supervised learning approaches to pre-training neural models for NLP, with recent successes including ELMo (Peters et al., 2018), GPT (Radford et al., 2018), and BERT (Devlin et al., 2019). Of late, the BERT model based on pre-training of a large transformer model (Vaswani et al., 2017) to encode bidirectional context has emerged as a dominant paradigm, thanks to …

Content truncated to 500 words