Unsupervised Labeled Parsing with Deep Inside-Outside Recursive Autoencoders

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1507–1512, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1507 Unsupervised Labeled Parsing with Deep Inside-Outside Recursive Autoencoders Andrew Drozdov, Pat Verga, Yi-Pei Chen, Mohit Iyyer, and Andrew McCallum College of Information and Computer Sciences University of Massachusetts Amherst {adrozdov, pat, yipeichen, miyyer, mccallum}@cs.umass.edu Abstract Understanding text often requires identifying meaningful constituent spans such as noun phrases and verb phrases. In this work, we show that we can effectively recover these types of labels using the learned phrase vec- tors from deep inside-outside recursive au- toencoders (DIORA). Specifically, we clus- ter span representations to induce span labels. Additionally, we improve the model’s labeling accuracy by integrating latent code learning into the training procedure. We evaluate this approach empirically through unsupervised la- beled constituency parsing. Our method out- performs ELMo and BERT on two versions of the Wall Street Journal (WSJ) dataset and is competitive to prior work that requires ad- ditional human annotations, improving over a previous state-of-the-art system that depends on ground-truth part-of-speech tags by 5 abso- lute F1 points (19% relative error reduction). 1 Introduction The deep inside-outside recursive autoencoder (Drozdov et al., 2019, DIORA) is part of a recent trend in fully unsupervised neural constituency parsers (Shen et al., 2018; Williams et al., 2018a; Htut et al., 2018; Shen et al., 2019; Kim et al., 2019). However, these works and nearly all previ- ous research (Klein and Manning, 2002; Seginer, 2007; Ponvert et al., 2011; Spitkovsky et al., 2013) have focused on unlabeled constituency parsing. In this paper, we instead focus on labeled con- stituency parsing for English. The small number of previous works that exist in this area suffer from substantial weaknesses: 1) the models depend on ground-truth part-of-speech tags, which are not al- ways available and known to boost constituency parsing scores (Kitaev and Klein, 2018), 2) none can simultaneously identify and label constituents (instead they typically depend on an external la- tent parser), and 3) they ignore sentences longer 0.7 ⋅ [ The cat ] [ drinks ] 0.2 0.9 0.2 { + Codebook 0.3 ⋅ Inside Pass 0.8 0.3 0.6 { + [ The ] [ cat drinks ] + Figure 1: The left half of this figure depicts the inside- pass of the DIORA model as described in (Drozdov et al., 2019). We are interested in clustering the learned vectors a(i, j) such that each span may be mapped to a phrase type. To enhance this clustering based approach, we augment the DIORA architecture with latent codes, shown in the right half of the figure. than ten tokens because previous latent parsers do not scale to longer sentences (Haghighi and Klein, 2006; Borensztajn and Zuidema, 2007; Reichart and Rappoport, 2008). Unlike previous work, we achieve strong results in unlabeled constituency parsing using a single model for both bracketing and labeling. Our ap- proach relies on clustering …

Content truncated to 500 words