Phrase-aware Unsupervised Constituency Parsing

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6406 - 6415 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Phrase-aware Unsupervised Constituency Parsing Xiaotao Gu1 Yikang Shen2,3 Jiaming Shen4 Jingbo Shang5 Jiawei Han1 1University of Illinois at Urbana-Champaign 5University of California San Diego 2Mila / Université de Montréal 3Pattern Recognition Center, WeChat AI, Tencent Inc 4Google Research 1{xiaotao2, hanj}@illinois.edu 5jshang@ucsd.edu 2,3yikang.shn@gmail.com 4jmshen@google.com Abstract Recent studies have achieved inspiring suc- cess in unsupervised grammar induction us- ing masked language modeling (MLM) as the proxy task. Despite their high accuracy in identifying low-level structures, prior arts tend to struggle in capturing high-level structures like clauses, since the MLM task usually only requires information from local context. In this work, we revisit LM-based constituency parsing from a phrase-centered perspective. In- spired by the natural reading process of hu- man readers, we propose to regularize the parser with phrases extracted by an unsuper- vised phrase tagger to help the LM model quickly manage low-level structures. For a bet- ter understanding of high-level structures, we propose a phrase-guided masking strategy for LM to emphasize more on reconstructing non- phrase words. We show that the initial phrase regularization serves as an effective bootstrap, and phrase-guided masking improves the iden- tification of high-level structures. Experiments on the public benchmark with two different backbone models demonstrate the effective- ness and generality of our method. 1 Introduction The hierarchical structure of natural language plays a key role in accurate language understanding, but can be unfortunately overlooked when text is treated as a plain sequence. To this end, consider- able efforts have been made in integrating structural inductive bias into neural language models (LM) (Shen et al., 2018b; Wang et al., 2019; Shen et al., 2020). Despite different implementations, the gen- eral idea is to first apply a parsing module to in- duce the soft grammar tree of the input text, and then incorporate the induced tree into an encoding model (e.g., Transformer (Vaswani et al., 2017)). The model is optimized in an unsupervised manner with masked language modeling (MLM) (Devlin et al., 2019) as a common proxy task. These models have shown inspiring success in inducing meaningful parsing trees without human annotation, but still face two challenging problems. Firstly, the parsing module is randomly initialized at the beginning of the training process. Subopti- mal initial parsing accuracy can lead to problematic structural constraints in the encoder model, and further influence the training process and final per- formance (Gimpel and Smith, 2012). Secondly, the token-level language modeling task encourages the model to focus on local structures, since the reconstruction of a masked word mainly relies on its local context. As a result, the learned model achieves high accuracy in local constituents, like noun phrases (NP), but significantly worse accu- racy in high-level, long-distance structures, such as subordinate clauses (SBAR) and prepositional phrases (PP). On the PTB dataset, the most recent structured language model (Shen et al., 2020) still falls behind neural …

Content truncated to 500 words