Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4832–4845, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4832 Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders Andrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O’Gorman, Mohit Iyyer, and Andrew McCallum College of Information and Computer Sciences University of Massachusetts Amherst {adrozdov, srongali, yipeichen, togorman, miyyer, mccallum} @cs.umass.edu Abstract The deep inside-outside recursive autoencoder (DIORA; Drozdov et al. 2019a) is a self- supervised neural model that learns to induce syntactic tree structures for input sentences without access to labeled training data. In this paper, we discover that while DIORA ex- haustively encodes all possible binary trees of a sentence with a soft dynamic program, its vector averaging approach is locally greedy and cannot recover from errors when comput- ing the highest scoring parse tree in bottom-up chart parsing. To fix this issue, we introduce S-DIORA, an improved variant of DIORA that encodes a single tree rather than a softly- weighted mixture of trees by employing a hard argmax operation and a beam at each cell in the chart. Our experiments show that through fine-tuning a pre-trained DIORA with our new algorithm, we improve the state of the art in unsupervised constituency parsing on the En- glish WSJ Penn Treebank by 2.2 − 6% F1, depending on the data used for fine-tuning. 1 Introduction Syntactic parse trees are valuable intermediate fea- tures for many NLP pipelines (He et al., 2018; Strubell et al., 2018), as a soft constraint (Rush and Collins, 2012), a hard constraint (Lee et al., 2019b), or in multi-task learning with syntactic scaffolds (Swayamdipta et al., 2018). Syntac- tic inductive bias can also improve generalization of deep learning models (Kuncoro et al., 2020). These results have motivated researchers to pur- sue unsupervised parsing, with the hope of train- ing syntax-dependent models on large amounts of data without annotation (Klein and Manning, 2002; Bod, 2006; Ponvert et al., 2011; Shen et al., 2019; Kim et al., 2019, inter alia). Of these models, we focus on the deep inside- outside recursive autoencoder (DIORA; Drozdov et al. 2019a). DIORA encodes sentences in a DIORA … … 1 2 S-DIORA DIORA 1 2 S-DIORA Figure 1: DIORA (top row) is sensitive to locally non- optimal decisions. By assigning a low weight to a po- tentially important subtree when recursively computing the vector for a target tree, it is difficult or impossible to recover and the important subtree is washed out (repre- sented in light gray). Our method, S-DIORA (bottom row) can recover from errors, and the desired tree ends up at the top of the beam in the right-most column. procedure resembling the inside-outside algorithm (Baker, 1979), which allows it to induce syntactic tree structures for input sentences without access to labeled training data, and achieves near state- of-the-art results on unsupervised constituency parsing. DIORA resembles pre-trained language models, such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), in that it …

Content truncated to 500 words