CRF Autoencoder for Unsupervised Dependency Parsing

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1638–1643 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics CRF Autoencoder for Unsupervised Dependency Parsing∗ Jiong Cai, Yong Jiang and Kewei Tu {caijiong,jiangyong, tukw}@shanghaitech.edu.cn School of Information Science and Technology ShanghaiTech University, Shanghai, China Abstract Unsupervised dependency parsing, which tries to discover linguistic dependency structures from unannotated data, is a very challenging task. Almost all previous work on this task focuses on learning gen- erative models. In this paper, we de- velop an unsupervised dependency pars- ing model based on the CRF autoencoder. The encoder part of our model is discrim- inative and globally normalized which al- lows us to use rich features as well as uni- versal linguistic priors. We propose an exact algorithm for parsing as well as a tractable learning algorithm. We evaluated the performance of our model on eight multilingual treebanks and found that our model achieved comparable performance with state-of-the-art approaches. 1 Introduction Unsupervised dependency parsing, which aims to discover syntactic structures in sentences from un- labeled data, is a very challenging task in natural language processing. Most of the previous work on unsupervised dependency parsing is based on generative models such as the dependency model with valence (DMV) introduced by Klein and Manning (2004). Many approaches have been proposed to enhance these generative models, for example, by designing advanced Bayesian priors (Cohen et al., 2008), representing dependencies with features (Berg-Kirkpatrick et al., 2010), and representing discrete tokens with continuous vec- tors (Jiang et al., 2016). Besides generative approaches, Grave and El- hadad (2015) proposed an unsupervised discrim- ∗This work was supported by the National Natural Sci- ence Foundation of China (61503248). inative parser. They designed a convex quadratic objective function under the discriminative clus- tering framework. By utilizing global features and linguistic priors, their approach achieves state- of-the-art performance. However, their approach uses an approximate parsing algorithm, which has no theoretical guarantee. In addition, the perfor- mance of the approach depends on a set of manu- ally specified linguistic priors. Conditional random field autoencoder (Ammar et al., 2014) is a new framework for unsupervised structured prediction. There are two components of this model: an encoder and a decoder. The en- coder is a globally normalized feature-rich CRF model predicting the conditional distribution of the latent structure given the observed structured input. The decoder of the model is a generative model generating a transformation of the struc- tured input from the latent structure. Ammar et al. (2014) applied the model to two sequential structured prediction tasks, part-of-speech induc- tion and word alignment and showed that by uti- lizing context information the model can achieve better performance than previous generative mod- els and locally normalized models. However, to the best of our knowledge, there is no previous work applying the CRF autoencoder to tasks with more complicated outputs such as tree structures. In this paper, we propose an unsupervised dis- criminative dependency parser based on the CRF autoencoder framework and provide …

Content truncated to 500 words