Second-Order Unsupervised Neural Dependency Parsing

Proceedings of the 28th International Conference on Computational Linguistics, pages 3911–3924 Barcelona, Spain (Online), December 8-13, 2020 3911 Second-Order Unsupervised Neural Dependency Parsing Songlin Yang1,2,3, Yong Jiang4, Wenjuan Han5, Kewei Tu1∗ 1School of Information Science and Technology, ShanghaiTech University 2Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences 3University of Chinese Academy of Sciences 4Alibaba DAMO Academy, Alibaba Group 5Department of Computer Science, National University of Singapore {yangsl,tukw}@shanghaitech.edu.cn yongjiang.jy@alibaba-inc.com dcshanw@nus.edu.sg Abstract Most of the unsupervised dependency parsers are based on first-order probabilistic generative models that only consider local parent-child information. Inspired by second-order supervised dependency parsing, we proposed a second-order extension of unsupervised neural dependency models that incorporate grandparent-child or sibling information. We also propose novel design of the neural parameterization and optimization methods of the dependency models. In second- order models, the number of grammar rules grows cubically with the increase of vocabulary size, making it difficult to train lexicalized models that may contain thousands of words. To circum- vent this problem while still benefiting from both second-order parsing and lexicalization, we use the agreement-based learning framework to jointly train a second-order unlexicalized model and a first-order lexicalized model. Experiments on multiple datasets show the effectiveness of our second-order models compared with recent state-of-the-art methods. Our joint model achieves a 10% improvement over the previous state-of-the-art parser on the full WSJ test set.1 1 Introduction Dependency parsing is a classical task in natural language processing. The head-dependent relations produced by dependency parsing can provide an approximation to the semantic relationship between words, which is useful in many downstream NLP tasks such as machine translation, information extrac- tion and question answering. Nowadays, supervised dependency parsers can reach a very high accuracy (Dozat and Manning, 2017; Zhang et al., 2020). Unfortunately, supervised parsing requires treebanks (annotated parse trees) for training, which are very expensive and time-consuming to build. On the other hand, unsupervised dependency parsing requires only unannotated corpora for training, though the accu- racy of unsupervised parsing still lags far behind that of supervised parsing. We focus on unsupervised dependency parsing in this paper. Most methods in the literature of unsupervised dependency parsing are based on the Dependency Model with Valence (DMV) (Klein and Manning, 2004), which is a probabilistic generative model. A main disadvantage of DMV and many of its extensions is that they lack expressiveness. The generation of a dependent token is only conditioned on its parent, the relative direction of the token to its parent, and whether its parent has already generated any child in this direction, hence ignoring other contex- tual information. To improve model expressiveness, researchers often turn to discriminative methods, which can incorporate more contextual information into the scoring or prediction of dependency arcs. For example, Grave and Elhadad (2015) uses the idea of disrciminative clustering, Cai et al. (2017) uses a discriminative parser in the CRF-autoencoder framework, and Li et al. (2018) uses an encoder-decoder framework that contains a discriminative transitioned-based parser. For DMV, Han et al. (2019) proposes the discriminative …

Content truncated to 500 words