Evaluating Universal Dependency Parser Recovery of Predicate Argument Structure via CompChain Analysis

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 116–128 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 116 Evaluating Universal Dependency Parser Recovery of Predicate Argument Structure via CompChain Analysis Sagar Indurkhya LIDS/IDSS, Dept. of EECS MIT 32 Vassar St. Cambridge, MA 02139 indurks@mit.edu Beracah Yankama LIDS/IDSS, Dept. of EECS MIT 32 Vassar St. Cambridge, MA 02139 beracah@mit.edu Robert C. Berwick LIDS/IDSS, Dept. of EECS MIT Room 32D-728 32 Vassar St. Cambridge, MA 02139 berwick@csail.mit.edu Abstract Accurate recovery of predicate-argument structure from a Universal Dependency (UD) parse is central to downstream tasks such as extraction of semantic roles or event represen- tations. This study introduces compchains, a categorization of the hierarchy of predicate de- pendency relations present within a UD parse. Accuracy of compchain classification serves as a proxy for measuring accurate recovery of predicate-argument structure from sentences with embedding. We analyzed the distri- bution of compchains in three UD English treebanks, EWT, GUM and LinES, revealing that these treebanks are sparse with respect to sentences with predicate-argument structure that includes predicate-argument embedding. We evaluated the CoNLL 2018 Shared Task UDPipe (v1.2) baseline (dependency parsing) models as compchain classifiers for the EWT, GUMS and LinES UD treebanks. Our results indicate that these three baseline models exhibit poorer performance on sentences with predicate-argument structure with more than one level of embedding; we used compchains to characterize the errors made by these parsers and present examples of erroneous parses produced by the parser that were identi- fied using compchains. We also analyzed the distribution of compchains in 58 non-English UD treebanks and then used compchains to evaluate the CoNLL’18 Shared Task baseline model for each of these treebanks. Our analysis shows that performance with respect to compchain classification is only weakly correlated with the official evaluation metrics (LAS, MLAS and BLEX). We identify gaps in the distribution of compchains in several of the UD treebanks, thus providing a roadmap for how these treebanks may be supplemented. We conclude by discussing how compchains provide a new perspective on the sparsity of training data for UD parsers, as well as the accuracy of the resulting UD parses. 1 Introduction The Universal Dependencies (UD) project (De Marneffe et al., 2014; Nivre et al., 2016) is a multilingual annotation scheme for dependency grammars that has gained wide usage (Zeman et al., 2017; Kong et al., 2017; Qi et al., 2020). To this extent, automatically identifying whether a dependency parse1 is correct or incorrect, as well as the potential source of such errors, becomes an important part of NLP pipelines. For example, such identification can prevent errors from propagating to downstream applications such as the identification of predicate-argument structure, involved in semantic role labeling and sentiment analysis.2 Furthermore, embedding of sentences within sentences, and in particular embedding of predicate-argument structures within one another, is one of the ways in which humans have the capability to generate an infinity of different <sentences, meaning> pairings, and so it is important to evaluate whether …

Content truncated to 500 words