Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5208–5222 July 9-14, 2023 ©2023 Association for Computational Linguistics Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers Jiaxi Li and Wei Lu StatNLP Research Group Singapore University of Technology and Design jiaxi_li@mymail.sutd.edu.sg, luwei@sutd.edu.sg Abstract Recent advancements in pre-trained language models (PLMs) have demonstrated that these models possess some degree of syntactic aware- ness. To leverage this knowledge, we propose a novel chart-based method for extracting parse trees from masked language models (LMs) without the need to train separate parsers. Our method computes a score for each span based on the distortion of contextual representations resulting from linguistic perturbations. We de- sign a set of perturbations motivated by the linguistic concept of constituency tests, and use these to score each span by aggregating the dis- tortion scores. To produce a parse tree, we use chart parsing to find the tree with the minimum score. Our method consistently outperforms previous state-of-the-art methods on English with masked LMs, and also demonstrates su- perior performance in a multilingual setting, outperforming the state of the art in 6 out of 8 languages. Notably, although our method does not involve parameter updates or extensive hy- perparameter search, its performance can even surpass some unsupervised parsing methods that require fine-tuning. Our analysis highlights that the distortion of contextual representation resulting from syntactic perturbation can serve as an effective indicator of constituency across languages.1 1 Introduction Constituency parsing is a fundamental task in natu- ral language processing (NLP) that involves uncov- ering the syntactic structure of a sentence by identi- fying the constituents it is composed of. While supervised constituency parsing methods neces- sitate the utilization of a labeled dataset contain- ing sentences and their corresponding constituency parses, unsupervised methods for generating syn- tax trees emerge because manual annotation is labor-intensive and requires specialized linguistic 1Our code is available at https://github.com/ jxjessieli/contextual-distortion-parser. they watched a film this afternoon watched they this afternoon a film <mask> <mask> <mask> a film watched they this afternoon , , (1) (2) (3) Figure 1: Example sentence and its constituency tree. We list perturbed sentences after substitution (1), decon- textualization (2), and movement (3). knowledge. One line of work for unsupervised constituency parsing involves designing an objec- tive function that enables the model to infer the hierarchical structure of language from the unan- notated text (Kim et al., 2019b,a; Drozdov et al., 2019; Yang et al., 2021). An alternative approach, known as Constituency Parse Extraction from Pre- trained Language Models (CPE-PLM), involves extracting parse trees from pre-trained language models without fine-tuning in an unsupervised man- ner (Kim et al., 2020; Wu et al., 2020; Kim et al., 2021). The main motivation for CPE-PLM is the assumption that pre-trained language models con- tain implicit syntactic knowledge learned during the pre-training stage. This knowledge can then be used to directly predict parse trees, eliminating the need for task-specific fine-tuning. While CPE- PLM systems have been shown …

Content truncated to 500 words