Unsupervised Neural Dependency Parsing

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 763–771, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Unsupervised Neural Dependency Parsing∗ Yong Jiang, Wenjuan Han and Kewei Tu {jiangyong,hanwj,tukw}@shanghaitech.edu.cn School of Information Science and Technology ShanghaiTech University, Shanghai, China Abstract Unsupervised dependency parsing aims to learn a dependency grammar from text anno- tated with only POS tags. Various features and inductive biases are often used to incorpo- rate prior knowledge into learning. One use- ful type of prior information is that there exist correlations between the parameters of gram- mar rules involving different POS tags. Pre- vious work employed manually designed fea- tures or special prior distributions to encode such information. In this paper, we propose a novel approach to unsupervised dependen- cy parsing that uses a neural model to predict grammar rule probabilities based on distribut- ed representation of POS tags. The distributed representation is automatically learned from data and captures the correlations between POS tags. Our experiments show that our approach outperforms previous approaches u- tilizing POS correlations and is competitive with recent state-of-the-art approaches on nine different languages. 1 Introduction Unsupervised structured prediction from data is an important problem in natural language processing, with applications in grammar induction, POS tag in- duction, word alignment and so on. Because the training data is unannotated in unsupervised struc- tured prediction, learning is very hard. In this pa- per, we focus on unsupervised dependency parsing, which aims to identify the dependency trees of sen- tences in an unsupervised manner. ∗This work was supported by the National Natural Science Foundation of China (61503248). Previous work on unsupervised dependency pars- ing is mainly based on the dependency model with valence (DMV) (Klein and Manning, 2004) and it- s extension (Headden III et al., 2009; Gillenwater et al., 2010). To effectively learn the DMV mod- el for better parsing accuracy, a variety of induc- tive biases and handcrafted features have been pro- posed to incorporate prior information into learning. One useful type of prior information is that there exist correlations between the parameters of gram- mar rules involving different POS tags. Cohen and Smith (2009; 2010) employed special prior distribu- tions to encourage learning of correlations between POS tags. Berg-Kirkpatrick et al. (2010) encoded the relations between POS tags using manually de- signed features. In this work, we propose a neural based ap- proach to unsupervised dependency parsing. We incorporate a neural model into the DMV model to predict grammar rule probabilities based on dis- tributed representation of POS tags. We learn the neural network parameters as well as the distribut- ed representations from data using the expectation- maximization algorithm. The correlations between POS tags are automatically captured in the learned POS embeddings and contribute to the improvemen- t of parsing accuracy. In particular, probabilities of grammar rules involving correlated POS tags are au- tomatically smoothed in our approach without the need for manual features or additional smoothing procedures. Our experiments show that on the Wall …

Content truncated to 500 words