Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders

Proceedings of NAACL-HLT 2019, pages 1129–1141 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1129 Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders Andrew Drozdov∗, Pat Verga∗, Mohit Yadav∗, Mohit Iyyer, and Andrew McCallum College of Information and Computer Sciences University of Massachusetts Amherst {adrozdov, pat, ymohit, miyyer, mccallum}@cs.umass.edu Abstract We introduce deep inside-outside recursive autoencoders (DIORA), a fully-unsupervised method for discovering syntax that simulta- neously learns representations for constituents within the induced tree. Our approach pre- dicts each word in an input sentence condi- tioned on the rest of the sentence and uses inside-outside dynamic programming to con- sider all possible binary trees over the sen- tence. At test time the CKY algorithm extracts the highest scoring parse. DIORA achieves a new state-of-the-art F1 in unsupervised binary constituency parsing (unlabeled) in two bench- mark datasets, WSJ and MultiNLI. 1 Introduction Syntactic parse trees are useful for downstream tasks such as relation extraction (Gamallo et al., 2012), semantic role labeling (Sutton and Mc- Callum, 2005; He et al., 2018), machine trans- lation (Aharoni and Goldberg, 2017; Eriguchi et al., 2017; Zaremoodi and Haffari, 2018), and text classification (Li and Roth, 2006; Tai et al., 2015). Traditionally, supervised parsers trained on datasets such as the Penn Treebank (Marcus et al., 1993) are used to obtain syntactic trees. However, the treebanks used to train these su- pervised parsers are typically small and restricted to the newswire domain. Unfortunately, models trained on newswire treebanks tend to perform considerably worse when applied to new types of data, and creating new domain specific treebanks with syntactic annotations is expensive and time- consuming. Motivated by the desire to address the limita- tions of supervised parsing and by the success of large-scale unsupervised modeling such as ELMo and BERT (Peters et al., 2018a; Devlin et al., ∗Equal contribution, randomly ordered. Under the current circumstances he says their scenario no longer seems unrealistic Figure 1: An unlabeled binary constituency parse from DIORA matching the ground truth. 2019), we propose a new deep learning method of unsupervised parser training that can extract both shallow parses (i.e., noun phrases or entities) and full syntactic trees from any domain or lan- guage automatically without requiring any labeled training data. In addition to producing parses, our model simultaneously builds representations for internal constituents that reflect syntactic and semantic regularities which can be leveraged by downstream tasks. Our model builds on existing work developing latent tree chart parsers (Socher et al., 2011b; Le and Zuidema, 2015; Yogatama et al., 2017; Mail- lard et al., 2017; Choi et al., 2018). These meth- ods produce representations for all internal nodes in the tree (cells in the chart), each generated as a soft weighting over all possible sub-trees (§2). Unfortunately, they still require sentence-level an- notations during training, as they are all trained to optimize a downstream task, typically natural lan- guage inference. To address these limitations, we present deep inside-outside recursive autoencoders (DIORA) which enable unsupervised discovery and repre- …

Content truncated to 500 words