A Neural Network Model for Low-Resource Universal Dependency Parsing

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 339–348, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Neural Network Model for Low-Resource Universal Dependency Parsing Long Duong,12 Trevor Cohn,1 Steven Bird,1 and Paul Cook3 1Department of Computing and Information Systems, University of Melbourne 2National ICT Australia, Victoria Research Laboratory 3Faculty of Computer Science, University of New Brunswick lduong@student.unimelb.edu.au {t.cohn,sbird}@unimelb.edu.au paul.cook@unb.ca Abstract Accurate dependency parsing requires large treebanks, which are only available for a few languages. We propose a method that takes advantage of shared structure across languages to build a mature parser using less training data. We propose a model for learning a shared “univer- sal” parser that operates over an inter- lingual continuous representation of lan- guage, along with language-specific map- ping components. Compared with super- vised learning, our methods give a con- sistent 8-10% improvement across several treebanks in low-resource simulations. 1 Introduction Dependency parsing is an important task for Nat- ural Language Processing (NLP) with application to text classification ( ¨Ozg¨ur and G¨ung¨or, 2010), relation extraction (Bunescu and Mooney, 2005), question answering (Cui et al., 2005), statistical machine translation (Xu et al., 2009), and sen- timent analysis (Socher et al., 2013). A mature parser normally requires a large treebank for train- ing, yet such resources are rarely available and are costly to build. Ideally, we would be able to construct a high quality parser with less training data, thereby enabling accurate parsing for low- resource languages. In this paper we formalize the dependency pars- ing task for a low-resource language as a domain adaptation task, in which a target resource-poor language treebank is treated as in-domain, while a much larger treebank in a high-resource lan- guage forms the out-of-domain data. In this way, we can apply well-understood domain adaptation techniques to the dependency parsing task. How- ever, a crucial requirement for domain adaptation is that the in-domain and out-of-domain data have compatible representations. In applying our ap- proach to data from several languages, we must learn such a cross-lingual representation. Here we frame this representation learning as part of a neural network training. The underlying hypoth- esis for the joint learning is that there are some shared-structures across languages that we can ex- ploit. This hypothesis is motivated by the excellent results of the cross-lingual application of unlexi- calised parsing (McDonald et al., 2011), whereby a delexicalized parser constructed on one language is applied directly to another language. Our approach works by jointly training a neu- ral network dependency parser to model the syn- tax in both a source and target language. Many of the parameters of the source and target language parsers are shared, except for a small handful of language-specific parameters. In this way, the in- formation can flow back and forth between lan- guages, allowing for the learning of a compatible cross-lingual syntactic representation, while also allowing the parsers to mutually correct one an- other’s errors. We include some language-specific components, in order to better model …

Content truncated to 500 words