Improving Constituency Parsing with Span Attention

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1691–1703 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1691 Improving Constituency Parsing with Span Attention Yuanhe Tian♥, Yan Song♠♥†, Fei Xia♥, Tong Zhang♦ ♥University of Washington ♠The Chinese University of Hong Kong (Shenzhen) ♥Shenzhen Research Institute of Big Data ♦The Hong Kong University of Science and Technology ♥{yhtian, fxia}@uw.edu ♠songyan@cuhk.edu.cn ♦tongzhang@ust.hk Abstract Constituency parsing is a fundamental and im- portant task for natural language understand- ing, where a good representation of contex- tual information can help this task. N-grams, which is a conventional type of feature for con- textual information, have been demonstrated to be useful in many tasks, and thus could also be beneficial for constituency parsing if they are appropriately modeled. In this pa- per, we propose span attention for neural chart- based constituency parsing to leverage n-gram information. Considering that current chart- based parsers with Transformer-based encoder represent spans by subtraction of the hidden states at the span boundaries, which may cause information loss especially for long spans, we incorporate n-grams into span representa- tions by weighting them according to their contributions to the parsing process. More- over, we propose categorical span attention to further enhance the model by weighting n- grams within different length categories, and thus benefit long-sentence parsing. Experi- mental results on three widely used benchmark datasets demonstrate the effectiveness of our approach in parsing Arabic, Chinese, and En- glish, where state-of-the-art performance is ob- tained by our approach on all of them.1 1 Introduction Constituency parsing, which aims to generate a structured syntactic parse tree for a given sentence, is one of the most fundamental tasks in natural language processing (NLP), and plays an impor- tant role in many downstream tasks such as re- lation extraction (Jiang and Diesner, 2019), nat- ural language inference (Chen et al., 2017), and machine translation (Ma et al., 2018). Recently, †Corresponding author. 1Our code and the best performing models are released at https://github.com/cuhksz-nlp/SAPar. Figure 1: The treelet of an example of the form “V+NP+PP”, where the “PP” should attach to the “V” (in green) rather than the “NP” (in red). neural parsers (Vinyals et al., 2015; Dyer et al., 2016; Stern et al., 2017; Kitaev et al., 2019) with- out using any grammar rules significantly outper- form conventional statistical grammar-based ones (Collins, 1997; Sagae and Lavie, 2005; Glaysher and Moldovan, 2006; Song and Kit, 2009), because neural networks, especially recurrent models (e.g, Bi-LSTM), are adept in capturing long range con- textual information, which is essential to modeling the entire sentence. Particularly, a significant boost on the performance of chart-based parsers is ob- served from some recent studies (Kitaev and Klein, 2018; Kitaev et al., 2019; Zhou and Zhao, 2019) that employ advanced text encoders (i.e., Trans- former, BERT, and XLNet), which further demon- strates the usefulness of contexts for parsing. In general, besides powerful encoders, other ex- tra information (such as pre-trained embeddings and extra syntactic information) can also provide useful contextual information and …

Content truncated to 500 words