Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation

Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation Goran Glavaˇs University of Mannheim Data and Web Science Group goran@informatik.uni-mannheim.de Ivan Vuli´c University of Cambridge Language Technology Lab iv250@cam.ac.uk Abstract Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU). The recent advent of end-to-end neural models, self-supervised via language model- ing (LM), and their success on a wide range of LU tasks, however, questions this belief. In this work, we empirically investigate the usefulness of supervised parsing for seman- tic LU in the context of LM-pretrained trans- former networks. Relying on the established fine-tuning paradigm, we first couple a pre- trained transformer with a biaffine parsing head, aiming to infuse explicit syntactic knowl- edge from Universal Dependencies treebanks into the transformer. We then fine-tune the model for LU tasks and measure the effect of the intermediate parsing training (IPT) on downstream LU task performance. Results from both monolingual English and zero-shot language transfer experiments (with intermedi- ate target-language parsing) show that explicit formalized syntax, injected into transformers through IPT, has very limited and inconsistent effect on downstream LU performance. Our results, coupled with our analysis of transform- ers’ representation spaces before and after in- termediate parsing, make a significant step to- wards providing answers to an essential ques- tion: how (un)availing is supervised parsing for high-level semantic natural language un- derstanding in the era of large neural models? 1 Introduction Structural analysis of sentences, based on a vari- ety of syntactic formalisms (Charniak, 1996; Tay- lor et al., 2003; De Marneffe et al., 2006; Hock- enmaier and Steedman, 2007; Nivre et al., 2016, 2020, inter alia), has been the beating heart of NLP pipelines for decades (Klein and Manning, 2003; Chen and Manning, 2014; Dozat and Manning, 2017; Kondratyuk and Straka, 2019), establishing rather strong common belief that high-level seman- tic language understanding (LU) crucially depends on explicit syntax. The unprecedented success of neural language learning models based on trans- former networks (Vaswani et al., 2017), trained on unlabeled corpora via language modeling (LM) objectives (Devlin et al., 2019; Liu et al., 2019b; Clark et al., 2020, inter alia) on a wide variety of LU tasks (Wang et al., 2018; Hu et al., 2020), how- ever, questions this widely accepted assumption. The question of necessity of supervised parsing for LU and NLP in general has been raised before. More than a decade ago, Bod (2007) questioned the superiority of supervised parsing over unsupervised induction of syntactic structures in the context of statistical machine translation. Nonetheless, the NLP community has since still managed to find sufficient evidence for the usefulness of explicit syntax in higher-level LU tasks (Levy and Gold- berg, 2014; Cheng and Kartsaklis, 2015; Bastings et al., 2017; Kasai et al., 2019; Zhang et al., 2019a, inter alia). However, we believe that the massive improvements brought about by the LM-pretrained transformers – unexposed to any explicit syntactic signal – warrant a renewed scrutiny of the utility of supervised parsing …

Content truncated to 500 words