How Much Syntactic Supervision is “Good Enough”?

Findings of the Association for Computational Linguistics: EACL 2023, pages 2300–2305 May 2-6, 2023 ©2023 Association for Computational Linguistics How Much Syntactic Supervision is “Good Enough”? Hiroshi Noji†∗ Artificial Intelligence Research Center AIST hiroshi.noji@aist.go.jp Yohei Oseki∗ Graduate School of Arts and Sciences University of Tokyo oseki@g.ecc.u-tokyo.ac.jp Abstract In this paper, we explore how much syntactic supervision is “good enough” to make language models (LMs) more human-like. Specifically, we propose the new method called syntactic ab- lation, where syntactic LMs, namely Recurrent Neural Network Grammars (RNNGs), are grad- ually ablated from full syntactic supervision to zero syntactic supervision (≈ unidirectional LSTM) by preserving NP, VP, PP, SBAR non- terminal symbols and the combinations thereof. The 17 ablated grammars are then evaluated via targeted syntactic evaluation on the SyntaxGym benchmark. The results of our syntactic abla- tion demonstrated that (i) the RNNG with zero syntactic supervision underperformed the RN- NGs with some syntactic supervision, (ii) the RNNG with full syntactic supervision underper- formed the RNNGs with less syntactic super- vision, and (iii) the RNNG with mild syntactic supervision achieved the best performance com- parable to the state-of-the-art GPT-2-XL. Those results may suggest that the “good enough” ap- proach to language processing seems to make LMs more human-like. 1 Introduction In the literature on targeted syntactic evaluation (Linzen et al., 2016; Marvin and Linzen, 2018), recurrent neural networks (RNNs) such as LSTMs have been demonstrated to implicitly learn syntac- tic structures of natural language (e.g., subject-verb agreement), despite the lack of explicit syntactic supervision (cf. Hewitt and Manning, 2019). More- over, those RNNs also turned out to benefit from explicit syntactic supervision. RNNs integrated with explicit syntactic supervision, namely Recur- rent Neural Network Grammars (RNNGs; Dyer et al. 2016), have received considerable attention for their cognitive plausibility and outperformed †Currently affiliated with LeapMind Inc.: noji@leapmind.io. *Denotes equal contribution. RNNs in not only targeted syntactic evaluation (Kuncoro et al., 2018; Wilcox et al., 2019) but also psychometric predictive power (Hale et al., 2018; Wilcox et al., 2020; Yoshida et al., 2021). However, despite the previous debate over the dichotomy between the presence and absence of syntactic supervision, how much syntactic super- vision is necessary and sufficient remains to be investigated. Especially, there are two potential reasons to believe that full syntactic supervision is suboptimal. Theoretically, full syntactic super- vision may override lexical heuristics implicitly learned with RNNs, where information on terminal symbols vanishes via recursive composition opera- tions (cf. Kuncoro et al., 2017). Empirically, full syntactic supervision seems to destroy the perfor- mance of long-distance dependencies, especially (pseudo-)cleft constructions, where both acceptable (e.g., What he did was prepare the meal.) and unac- ceptable (e.g., *What he ate was prepare the meal.) sentences share the exactly same syntactic structure (Figure 1) and should be distinguished via lexical heuristics alone (cf. Noji and Oseki, 2021). There- fore, it is reasonable to hypothesize that optimal syntactic supervision lies somewhere between full and zero syntactic supervision in order to balance syntactic structures and lexical heuristics. Intu- itively speaking, …

Content truncated to 500 words