Revisiting Higher-Order Dependency Parsers

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8795–8800 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8795 Revisiting Higher-Order Dependency Parsers Erick Fonseca Instituto de Telecomunicac¸˜oes Lisbon, Portugal erick.fonseca@lx.it.pt Andr´e F. T. Martins Instituto de Telecomunicac¸˜oes & Unbabel Lisbon, Portugal andre.t.martins@tecnico.ulisboa.pt Abstract Neural encoders have allowed dependency parsers to shift from higher-order structured models to simpler first-order ones, making de- coding faster and still achieving better accu- racy than non-neural parsers. This has led to a belief that neural encoders can implic- itly encode structural constraints, such as sib- lings and grandparents in a tree. We tested this hypothesis and found that neural parsers may benefit from higher-order features, even when employing a powerful pre-trained en- coder, such as BERT. While the gains of higher-order features are small in the presence of a powerful encoder, they are consistent for long-range dependencies and long sentences. In particular, higher-order models are more ac- curate on full sentence parses and on the ex- act match of modifier lists, indicating that they deal better with larger, more complex struc- tures. 1 Introduction Before the advent of neural networks in NLP, de- pendency parsers relied on higher-order features to better model sentence structure (McDonald and Pereira, 2006; Carreras, 2007; Koo and Collins, 2010; Martins et al., 2013, inter alia). Common choices for such features were siblings (a head word and two modifiers) and grandparents (a head word, its own head and a modifier). Kiperwasser and Goldberg (2016) showed that even without higher order features, a parser with an RNN encoder could achieve state-of-the-art re- sults. This led folk wisdom to suggest that model- ing higher-order features in a neural parser would not bring additional advantages, and nearly all re- cent research on dependency parsing was restricted to first-order models (Dozat and Manning, 2016; Smith et al., 2018a). Kulmizev et al. (2019) fur- ther reinforced this belief comparing transition and graph-based decoders (but none of which higher order); Falenska and Kuhn (2019) suggested that higher-order features become redundant because the parsing models encode them implicitly. However, there is some evidence that neural parsers still benefit from structure modeling. Zhang et al. (2019) showed that a parser trained with a global structure loss function has higher accuracy than when trained with a local objective (i.e., learn- ing the head of each word independently). Falen- ska and Kuhn (2019) examined the impact of con- secutive sibling features in a neural dependency parser. While they found mostly negative results in a transition-based setting, a graph-based parser still showed significant gains on two out of 10 tree- banks. In this paper, we test rigorously the hypothesis of the utility of second-order features. In particu- lar, we experiment with consecutive sibling and grandparent features in a non-projective, graph- based dependency parser. We found that without a pretrained encoder, these features are only useful for large treebanks; however, when using BERT, they can improve performance on most treebanks we tested on — especially true …

Content truncated to 500 words