The Limitations of Limited Context for Constituency Parsing

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2675–2687 August 1–6, 2021. ©2021 Association for Computational Linguistics 2675 The Limitations of Limited Context for Constituency Parsing Yuchen Li Carnegie Mellon University yuchenl4@andrew.cmu.edu Andrej Risteski Carnegie Mellon University aristesk@andrew.cmu.edu Abstract Incorporating syntax into neural approaches in NLP has a multitude of practical and scientific benefits. For instance, a language model that is syntax-aware is likely to be able to produce better samples; even a discriminative model like BERT with a syntax module could be used for core NLP tasks like unsupervised syntactic parsing. Rapid progress in recent years was arguably spurred on by the empirical success of the Parsing-Reading-Predict architecture of (Shen et al., 2018a), later simplified by the Order Neuron LSTM of (Shen et al., 2019). Most notably, this is the first time neural ap- proaches were able to successfully perform un- supervised syntactic parsing (evaluated by var- ious metrics like F-1 score). However, even heuristic (much less fully math- ematical) understanding of why and when these architectures work is lagging severely behind. In this work, we answer representa- tional questions raised by the architectures in (Shen et al., 2018a, 2019), as well as some transition-based syntax-aware language mod- els (Dyer et al., 2016): what kind of syntac- tic structure can current neural approaches to syntax represent? Concretely, we ground this question in the sandbox of probabilistic context-free-grammars (PCFGs), and identify a key aspect of the representational power of these approaches: the amount and direc- tionality of context that the predictor has ac- cess to when forced to make parsing deci- sion. We show that with limited context (either bounded, or unidirectional), there are PCFGs, for which these approaches cannot represent the max-likelihood parse; conversely, if the context is unlimited, they can represent the max-likelihood parse of any PCFG. 1 Introduction Neural approaches have been steadily making their way to NLP in recent years. By and large however, the neural techniques that have been scaled-up the most and receive widespread usage do not explic- itly try to encode discrete structure that is natural to language, e.g. syntax. The reason for this is perhaps not surprising: neural models have largely achieved substantial improvements in unsupervised settings, BERT (Devlin et al., 2019) being the de- facto method for unsupervised pre-training in most NLP settings. On the other hand unsupervised syn- tactic tasks, e.g. unsupervised syntactic parsing, have long been known to be very difficult tasks (Htut et al., 2018). However, since incorporating syntax has been shown to improve language model- ing (Kim et al., 2019b) as well as natural language inference (Chen et al., 2017; Pang et al., 2019; He et al., 2020), syntactic parsing remains important even in the current era when large pre-trained mod- els, like BERT (Devlin et al., 2019), are available. Arguably, the breakthrough works in unsuper- vised constituency parsing in a neural manner were (Shen et al., 2018a, 2019), achieving F1 scores …

Content truncated to 500 words