BERT-Proof Syntactic Structures: Investigating Errors in Discontinuous Constituency Parsing

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3259–3272 August 1–6, 2021. ©2021 Association for Computational Linguistics 3259 BERT-Proof Syntactic Structures: Investigating Errors in Discontinuous Constituency Parsing Maximin Coavoux Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG maximin.coavoux@univ-grenoble-alpes.fr Abstract The combined use of neural scoring systems and BERT fine-tuning has led to very high results in many natural language processing (NLP) tasks. These high results raise two im- portant questions about the contribution and the limitations of pretrained-language models: (i) what are the remaining errors in the best- performing systems? (ii) what are the types of test examples where pretrained language mod- els help the most? In this paper, we investigate both questions for the task of English discon- tinuous constituency parsing on the Penn Tree- bank, for which recent models obtain close to 95 F1 score. To do so, we propose two meth- ods for automatically analysing the errors of discontinuous parser. First, we annotate and release a test-suite focused on the syntactic phenomena responsible for discontinuities in the Penn Treebank, enabling us to obtain a per-phenomenon evaluation of a parser’s out- put. Second, we extend the Berkeley Parser Analyser — a tool that classifies parsing er- rors according to predefined structural patterns —, to discontinuous trees. We apply both methods to characterize errors of a state-of-the- art transition-based discontinuous parser, and to provide an overview of the contribution of BERT to this task. 1 Introduction Discontinous constituency trees are phrase-based syntactic representations where the constraint stat- ing that a single phrase must yield a continuous sequence of tokens is lifted. Such representations are well-suited for modelling long-range dependen- cies, that typically arise for some syntactic phe- nomena, such as extractions or scrambling. For example, Figure 1 presents a discontinous VP mod- elling the relationship between the verb want and its extracted complement How many. In constituency treebanks, these long range de- pendencies are sometimes represented with typed SBARQ ┌────┴─────────┐ S │ ┌────┼────┐ │ VP │ │ │ ┌────┴─── │ ── │ ───┐ │ WHNP │ │ │ │ ┌───┴────┐ │ │ │ │ WRB JJ VBP NP VB . │ │ │ │ │ │ How many do you want ? Figure 1: Discontinuous constituency tree. empty categories (traces), coindexed with a dis- placed phrase (Marcus et al., 1993). However, pro- jective parsers usually ignore them.1 Indeed, the norm in Penn Treebank constituency parsing is to preprocess empty categories out of the corpus, leav- ing out important linguistic information. Formally, discontinuous constituency trees inter- pret as derivations from mildly context-sensitive grammar formalisms, such as linear context-free rewriting systems (Vijay-Shanker et al., 1987, LCFRS) or multiple context-free grammars (Seki et al., 1991, MCFG). As a result, exact parsing of discontinuous structures has high computational complexity. For example, CKY-style parsing of an LCFRS is O(n3f) in time (Kallmeyer, 2010), where f is the fan-out of the grammar: the maxi- mum number of spans in a grammar rule.2 The current state of the art for English discon- tinuous constituency parsing …

Content truncated to 500 words