Enhancing Unsupervised Generative Dependency Parser with Contextual Information

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5315–5325 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5315 Enhancing Unsupervised Generative Dependency Parser with Contextual Information Wenjuan Han, Yong Jiang and Kewei Tu∗ {hanwj,jiangyong,tukw}@shanghaitech.edu.cn School of Information Science and Technology ShanghaiTech University, Shanghai, China Abstract Most of the unsupervised dependency parsers are based on probabilistic generative models that learn the joint distribution of the given sentence and its parse. Probabilistic gener- ative models usually explicit decompose the desired dependency tree into factorized gram- mar rules, which lack the global features of the entire sentence. In this paper, we pro- pose a novel probabilistic model called dis- criminative neural dependency model with va- lence (D-NDMV) that generates a sentence and its parse from a continuous latent repre- sentation, which encodes global contextual in- formation of the generated sentence. We pro- pose two approaches to model the latent rep- resentation: the first deterministically sum- marizes the representation from the sentence and the second probabilistically models the representation conditioned on the sentence. Our approach can be regarded as a new type of autoencoder model to unsupervised de- pendency parsing that combines the benefits of both generative and discriminative tech- niques. In particular, our approach breaks the context-free independence assumption in pre- vious generative approaches and therefore be- comes more expressive. Our extensive experi- mental results on seventeen datasets from var- ious sources show that our approach achieves competitive accuracy compared with both gen- erative and discriminative state-of-the-art un- supervised dependency parsers. 1 Introduction Dependency parsing is a very important task in natural language processing. The dependency re- lations identified by dependency parsing convey syntactic information useful in subsequent appli- cations such as semantic parsing, information ex- traction, and question answering. In this paper, we ∗Corresponding author focus on unsupervised dependency parsing, which aims to induce a dependency parser from training sentences without gold parse annotation. Most previous approaches to unsupervised de- pendency parsing are based on probabilistic gener- ative models, for example, the Dependency Model with Valence (DMV) (Klein and Manning, 2004) and its extensions (Cohen and Smith, 2009; Head- den III et al., 2009; Cohen and Smith, 2010; Berg- Kirkpatrick et al., 2010; Gillenwater et al., 2010; Jiang et al., 2016). A disadvantage of such ap- proaches comes from the context-freeness of de- pendency grammars, a strong independence as- sumption that limits the information available in determining how likely a dependency is between two words in a sentence. In DMV, the probability of a dependency is computed from only the head and child tokens, the dependency direction, and the number of dependencies already connected from the head token. Additional information used for computing dependency probabilities in later work is also limited to local morpho-syntactic fea- tures such as word forms, lemmas and categories (Berg-Kirkpatrick et al., 2010), which does not break the context-free assumption. More recently, researchers have started to uti- lize discriminative methods in unsupervised de- pendency parsing based on the …

Content truncated to 500 words