Dependency Grammar Induction with Neural Lexicalization and Big Training Data

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1683–1688 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Dependency Grammar Induction with Neural Lexicalization and Big Training Data∗ Wenjuan Han, Yong Jiang and Kewei Tu {hanwj, jiangyong ,tukw}@shanghaitech.edu.cn School of Information Science and Technology ShanghaiTech University, Shanghai, China Abstract We study the impact of big models (in terms of the degree of lexicalization) and big data (in terms of the training cor- pus size) on dependency grammar induc- tion. We experimented with L-DMV, a lexicalized version of Dependency Model with Valence (Klein and Manning, 2004) and L-NDMV, our lexicalized extension of the Neural Dependency Model with Va- lence (Jiang et al., 2016). We find that L-DMV only benefits from very small de- grees of lexicalization and moderate sizes of training corpora. L-NDMV can bene- fit from big training data and lexicaliza- tion of greater degrees, especially when enhanced with good model initialization, and it achieves a result that is competitive with the current state-of-the-art. 1 Introduction Grammar induction is the task of learning a gram- mar from a set of unannotated sentences. In the most common setting, the grammar is unlexical- ized with POS tags being the tokens, and the train- ing data is the WSJ10 corpus (the Wall Street Journal corpus with sentences no longer than 10 words) containing no more than 6,000 training sentences (Cohen et al., 2008; Berg-Kirkpatrick et al., 2010; Tu and Honavar, 2012). Lexicalized grammar induction aims to incor- porate lexical information into the learned gram- mar to increase its representational power and im- prove the learning accuracy. The most straight- forward approach to encoding lexical informa- tion is full lexicalization (Pate and Johnson, 2016; Spitkovsky et al., 2013). A major problem with ∗This work was supported by the National Natural Sci- ence Foundation of China (61503248). full lexicalization is that the grammar becomes much larger and thus learning is more data de- manding. To mitigate this problem, Headden et al. (2009) and Blunsom and Cohn (2010) used par- tial lexicalization in which infrequent words are replaced by special symbols or their POS tags. Another straightforward way to mitigate the data scarcity problem of lexicalization is to use training corpora larger than the standard WSJ corpus. For example, Pate and Johnson (2016) used two large corpora containing more than 700k sentences; Marecek and Straka (2013) utilized a very large corpus based on Wikipedia in learning an unlexi- calized dependency grammar. Finally, smoothing techniques can be used to reduce the negative im- pact of data scarcity. One example is Neural DMV (NDMV) (Jiang et al., 2016) which incorporates neural networks into DMV and can automatically smooth correlated grammar rule probabilities. Inspired by this background, we conduct a sys- tematic study regarding the impact of the degree of lexicalization and the training data size on the accuracy of grammar induction approaches. We experimented with a lexicalized version of Depen- dency Model with Valence (L-DMV) (Klein and Manning, 2004) and our lexicalized …

Content truncated to 500 words