StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7196–7209 August 1–6, 2021. ©2021 Association for Computational Linguistics 7196 StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling Yikang Shen∗ Mila/Universit´e de Montr´eal Yi Tay Google Research Che Zheng Google Research Dara Bahri Google Research Donald Metzler Google Research Aaron Courville Mila/Universit´e de Montr´eal Abstract There are two major classes of natural lan- guage grammars — the dependency grammar that models one-to-one correspondences be- tween words and the constituency grammar that models the assembly of one or several corresponded words. While previous unsuper- vised parsing methods mostly focus on only in- ducing one class of grammars, we introduce a novel model, StructFormer, that can simulta- neously induce dependency and constituency structure. To achieve this, we propose a new parsing framework that can jointly generate a constituency tree and dependency graph. Then we integrate the induced dependency relations into the transformer, in a differentiable man- ner, through a novel dependency-constrained self-attention mechanism. Experimental re- sults show that our model can achieve strong results on unsupervised constituency parsing, unsupervised dependency parsing, and masked language modeling at the same time. 1 Introduction Human languages have a rich latent structure. This structure is multifaceted, with the two major classes of grammar being dependency and constituency structures. There has been an exciting breath of recent work targeted at learning this structure in a data-driven unsupervised fashion (Klein and Man- ning, 2002; Klein, 2005; Le and Zuidema, 2015; Shen et al., 2018c; Kim et al., 2019a). The core principle behind recent methods that induce struc- ture from data is simple - provide an inductive bias that is conducive for structure to emerge as a byproduct of some self-supervised training, e.g., language modeling. To this end, a wide range of models have been proposed that are able to success- fully learn grammar structures (Shen et al., 2018a,c; ∗ Corresponding author: yikang.shn@gmail.ca. Work done while interning at Google Reseach. Wang et al., 2019; Kim et al., 2019b,a). However, most of these works focus on inducing either con- stituency or dependency structures alone. In this paper, we make two important techni- cal contributions. First, we introduce a new neu- ral model, StructFormer, that is able to simultane- ously induce both dependency structure and con- stituency structure. Specifically, our approach aims to unify latent structure induction of different types of grammar within the same framework. Second, StructFormer is able to induce dependency struc- tures from raw data in an end-to-end unsupervised fashion. Most existing approaches induce depen- dency structures from other syntactic information like gold POS tags (Klein and Manning, 2004; Co- hen and Smith, 2009; Jiang et al., 2016). Previous works, having trained from words alone, often re- quires additional information, like pre-trained word clustering (Spitkovsky et al., 2011), pre-trained word embedding (He et al., 2018), acoustic cues (Pate and Goldwater, 2013), or annotated data from related languages (Cohen et al., …

Content truncated to 500 words