Deep Inside-outside Recursive Autoencoder with All-span Objective

Proceedings of the 28th International Conference on Computational Linguistics, pages 3610–3615 Barcelona, Spain (Online), December 8-13, 2020 3610 Deep Inside-outside Recursive Autoencoder with All-span Objective Ruyue Hong†, Jiong Cai†⋄, Kewei Tu†⋄∗ †School of Information Science and Technology, ShanghaiTech University ⋄Shanghai Engineering Research Center of Intelligent Vision and Imaging Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences {hongry,caijiong,tukw}@shanghaitech.edu.cn Abstract Deep inside-outside recursive autoencoder (DIORA) is a neural-based model designed for unsu- pervised constituency parsing. During its forward computation, it provides phrase and contextual representations for all spans in the input sentence. By utilizing the contextual representation of each leaf-level span, the span of length 1, to reconstruct the word inside the span, the model is trained without labeled data. In this work, we extend the training objective of DIORA by making use of all spans instead of only leaf-level spans. We test our new training objective on datasets of two languages: English and Japanese, and empirically show that our method achieves improvement in parsing accuracy over the original DIORA. 1 Introduction Constituency parsing produces constituent parse trees that can be useful for downstream tasks, such as text classification (Tai et al., 2015), machine translation (Eriguchi et al., 2017; Wang et al., 2018) and semantic role labelling (Gildea and Palmer, 2002; He et al., 2017). Although supervised neural methods have very high accuracy, they require annotated data which can be very limited for low-resource languages and domains. Unsupervised constituency parsing is a task aiming to learn a constituency parser from text without annotated parse trees. There is a recent trend of solving this task with neural approaches (Shen et al., 2018; Kim et al., 2019a; Shen et al., 2019; Kim et al., 2019b; Drozdov et al., 2019). DIORA (Drozdov et al., 2019) is one of these models. It mimics the inside-outside algorithm (Baker, 1979) to build an inside chart and an outside chart for each input sentence. Each cell in these charts represents a representation of the inside text or the outside context of a span of the sentence. DIORA is trained by using the contextual representation of every single word to predict the word itself, which can be seen as a reconstruction procedure of the input sentence. The original training objective of DIORA only takes the n leaf-level spans into account for an n-word sentence. Since such a sentence has (n2 + n)/2 spans in total, we suppose that utilizing the rest of the spans during training can help the model learn better. Thus we propose a new training objective for DIORA that takes into account the reconstruction of all spans. We design weighting strategies to balance the influence from spans and sentences of different lengths in the new objective function. In the experiments, we train and test DIORA with our new objective function on datasets of two languages, English and Japanese. For both languages, we show empirically that the new training objective improves the performance of unsupervised constituency parsing. 2 Approach …

Content truncated to 500 words