Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 5398–5408 October 12–17, 2022. 5398 Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Models Taeuk Kim Dept. of Computer Science & Dept. of Artificial Intelligence Hanyang University, Seoul, South Korea kimtaeuk@hanyang.ac.kr Abstract Constituency Parse Extraction from Pre-trained Language Models (CPE-PLM) is a recent paradigm that attempts to induce constituency parse trees relying only on the internal knowl- edge of pre-trained language models. While attractive in the perspective that similar to in-context learning, it does not require task- specific fine-tuning, the practical effectiveness of such an approach still remains unclear, ex- cept that it can function as a probe for in- vestigating language models’ inner workings. In this work, we mathematically reformulate CPE-PLM and propose two advanced ensem- ble methods tailored for it, demonstrating that the new parsing paradigm can be competitive with common unsupervised parsers by intro- ducing a set of heterogeneous PLMs combined using our techniques. Furthermore, we explore some scenarios where the trees generated by CPE-PLM are practically useful. Specifically, we show that CPE-PLM is more effective than typical supervised parsers in few-shot settings. 1 Introduction With the increasing interest in the inner workings of pre-trained language models (PLMs; Devlin et al. (2019); Liu et al. (2019); Radford et al. (2019); Conneau et al. (2020)),1 much work that attempts to explore the inherent knowledge embedded in the models has been recently proposed. One of the main topics in this direction is to reveal whether PLMs understand syntactic knowledge of human language, usually represented as parse trees. While a line of work (Hewitt and Manning (2019); Chi et al. (2020)) has investigated the existence of syn- tax in PLMs via structural probes with supervision from gold-standard parse trees, some studies (Kim et al., 2020, 2021; Wu et al., 2020) have found that 1We use the term pre-trained language models (PLMs) to refer to the models that are based on Transformer (Vaswani et al., 2017) and pre-trained in a self-supervised manner, e.g., BERT (Devlin et al., 2019) and its variants. one can extract reasonable parse structures directly from the patterns presented in PLMs’ hidden rep- resentations or attention distributions even without extra fine-tuning. In other words, the studies have shown that PLMs implicitly store their understand- ing of syntactic knowledge in their parameters, and that such information can be easily reformulated into syntactic trees with almost no additional cost. Although the aforementioned approach, dubbed Constituency Parse Extraction from Pre-trained Language Models (CPE-PLM; Kim et al. (2021)), is undoubtedly a useful tool with many analytic uses (Rogers et al., 2020), it still remains a research question whether this algorithm can also work for practical purposes. For instance, as CPE-PLM is free from fine-tuning of PLMs, it may be appealing in few-shot settings, akin to in-context learning for natural language understanding (Brown et al., 2020). Moreover, there exists a potential that the new parsing paradigm can substitute the role of supervised or unsupervised parsers for the case where an …

Content truncated to 500 words