An Empirical Comparison of Unsupervised Constituency Parsing Methods

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3278–3283 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3278 An Empirical Comparison of Unsupervised Constituency Parsing Methods∗ Jun Li⋄, Yifan Cao⋄, Jiong Cai⋄, Yong Jiang†, and Kewei Tu⋄ ⋄School of Information Science and Technology, ShanghaiTech University, Shanghai, China ⋄Shanghai Engineering Research Center of Intelligent Vision and Imaging, Shanghai, China †DAMO Academy, Alibaba Group {lijun2, caoyf, caijiong, tukw}@shanghaitech.edu.cn yongjiang.jy@alibaba-inc.com Abstract Unsupervised constituency parsing aims to learn a constituency parser from a training corpus without parse tree annotations. While many methods have been proposed to tackle the problem, including statistical and neural methods, their experimental results are often not directly comparable due to discrepancies in datasets, data preprocessing, lexicalization, and evaluation metrics. In this paper, we first examine experimental settings used in previ- ous work and propose to standardize the set- tings for better comparability between meth- ods. We then empirically compare several existing methods, including decade-old and newly proposed ones, under the standardized settings on English and Japanese, two lan- guages with different branching tendencies. We find that recent models do not show a clear advantage over decade-old models in our ex- periments. We hope our work can provide new insights into existing methods and facilitate fu- ture empirical evaluation of unsupervised con- stituency parsing. 1 Introduction Unsupervised constituency parsing, a task in the area of grammar induction, aims to learn a con- stituency parser from a training corpus without parse tree annotations. While research on unsuper- vised constituency parsing has a long history (Car- roll and Charniak, 1992; Pereira and Schabes, 1992; Stolcke and Omohundro, 1994), recently there is a resurgence of interest in this task and several ap- proaches based on neural networks have been pro- posed that achieve impressive performance (Shen et al., 2018; Drozdov et al., 2019; Shen et al., 2019; Kim et al., 2019b,a; Jin et al., 2019). ∗ This work was supported by the National Natural Sci- ence Foundation of China (61976139). Kewei Tu is the corre- sponding author. With the recent development in research of unsu- pervised constituency parsing, however, the prob- lem of lacking a unified experimental setting be- gins to emerge, which makes empirical compari- son between different approaches difficult. First of all, although almost all previous approaches are evaluated on the Penn Treebank (Marcus and Marcinkiewicz, 1993), they differ in how they pre- process the training data, with respect to the sen- tence length limit, punctuation removal, vocabu- lary pruning, and so on. For example, non-neural methods such as Constituent Context Model (CCM) (Klein and Manning, 2002) are trained on short sen- tences, while modern neural based methods such as Parsing-Reading-Predict Network (PRPN) (Shen et al., 2018; Htut et al., 2018) do not impose any limit on sentence length. Furthermore, existing approaches also differ in their evaluation metrics, with respect to the meth- ods of computing averages, counting trivial spans, and so on. The evaluation results of the same ap- proach using different metrics can …

Content truncated to 500 words