An Empirical Investigation of Structured Output Modeling for Graph-based Neural Dependency Parsing

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5592–5598 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5592 An Empirical Investigation of Structured Output Modeling for Graph-based Neural Dependency Parsing Zhisong Zhang, Xuezhe Ma, Eduard Hovy Language Technologies Institute, Carnegie Mellon University {zhisongz,xuezhem}@cs.cmu.edu, hovy@cmu.edu Abstract In this paper, we investigate the aspect of structured output modeling for the state-of- the-art graph-based neural dependency parser (Dozat and Manning, 2017). With evalua- tions on 14 treebanks, we empirically show that global output-structured models can gen- erally obtain better performance, especially on the metric of sentence-level Complete Match. However, probably because neural models al- ready learn good global views of the inputs, the improvement brought by structured output modeling is modest. 1 Introduction In the past few years, dependency parsers, equipped with neural network models, have led to impressive empirical successes on parsing accu- racy (Chen and Manning, 2014; Weiss et al., 2015; Dyer et al., 2015; Andor et al., 2016; Kiperwasser and Goldberg, 2016; Kuncoro et al., 2016; Dozat and Manning, 2017; Ma et al., 2018). Among them, the deep-biaffine attentional parser (BiAF) (Dozat and Manning, 2017) has stood out for its simplicity and effectiveness. BiAF adopts a simple bi-directional LSTM neural architec- ture (Ma and Hovy, 2016; Kiperwasser and Gold- berg, 2016) with the first-order graph parsing al- gorithm (McDonald et al., 2005a,b). Simple as it appears to be, BiAF has led to several record- breaking performences in multiple treebanks and languages (Dozat et al., 2017). In their pioneering work, besides the neural ar- chitecture, Dozat and Manning (2017) adopt a simple head-selection training object (Zhang et al., 2017) by regarding the original structured predic- tion task as an head-classification task in train- ing. Although practically this simplification works well, there are still problems with it. Due to lo- cal normalization in the training objective (see §2.2), no global tree-structured information can be back-propagated during training. This can lead to the discrepancy between training and testing, since during testing, the MST (Maximum Span- ning Tree) algorithm (McDonald et al., 2005b) is used to ensure valid tree structures. This prob- lem raises concerns about the structured output layer. Several previous neural graph parsers uti- lized structured techniques (Pei et al., 2015; Kiper- wasser and Goldberg, 2016; Zhang et al., 2016; Wang and Chang, 2016; Ma and Hovy, 2017), but their neural architectures might not be competitive to the current state-of-the-art BiAF parsing model. In this paper, building upon the BiAF based neural architecture, we empirically investigate the effec- tiveness of utilizing classical structured prediction techniques of output modeling for graph-based neural dependency parsing. We empirically show that structured output modeling can obtain better performance, especially on the the sentence-level metrics. However, the improvements are modest, probably because neural models make the problem easier to solve locally. 2 Output Modeling In structured prediction tasks, a structured output y is predicted given an input x. We refer to the en- coding of the x …

Content truncated to 500 words