Diversity in Spectral Learning for Natural Language Parsing

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1868–1878, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Diversity in Spectral Learning for Natural Language Parsing Shashi Narayan and Shay B. Cohen School of Informatics University of Edinburgh Edinburgh, EH8 9LE, UK {snaraya2,scohen}@inf.ed.ac.uk Abstract We describe an approach to create a di- verse set of predictions with spectral learn- ing of latent-variable PCFGs (L-PCFGs). Our approach works by creating multiple spectral models where noise is added to the underlying features in the training set before the estimation of each model. We describe three ways to decode with mul- tiple models. In addition, we describe a simple variant of the spectral algorithm for L-PCFGs that is fast and leads to compact models. Our experiments for natural lan- guage parsing, for English and German, show that we get a significant improve- ment over baselines comparable to state of the art. For English, we achieve the F1 score of 90.18, and for German we achieve the F1 score of 83.38. 1 Introduction It has been long identified in NLP that a diverse set of solutions from a decoder can be reranked or re- combined in order to improve the accuracy in var- ious problems (Henderson and Brill, 1999). Such problems include machine translation (Macherey and Och, 2007), syntactic parsing (Charniak and Johnson, 2005; Sagae and Lavie, 2006; Fossum and Knight, 2009; Zhang et al., 2009; Petrov, 2010; Choe et al., 2015) and others (Van Halteren et al., 2001). The main argument behind the use of such a di- verse set of solutions (such as k-best list of parses for a natural language sentence) is the hope that each solution in the set is mostly correct. There- fore, recombination or reranking of solutions in that set will further optimize the choice of a solu- tion, combining together the information from all solutions. In this paper, we explore another angle for the use of a set of parse tree predictions, where all pre- dictions are made for the same sentence. More specifically, we describe techniques to exploit di- versity with spectral learning algorithms for natu- ral language parsing. Spectral techniques and the method of moments have been recently used for various problems in natural language processing, including parsing, topic modeling and the deriva- tion of word embeddings (Luque et al., 2012; Co- hen et al., 2013; Stratos et al., 2014; Dhillon et al., 2015; Rastogi et al., 2015; Nguyen et al., 2015; Lu et al., 2015). Cohen et al. (2013) showed how to estimate an L-PCFG using spectral techniques, and showed that such estimation outperforms the expectation- maximization algorithm (Matsuzaki et al., 2005). Their result still lags behind state of the art in natu- ral language parsing, with methods such as coarse- to-fine (Petrov et al., 2006). We further advance the accuracy of natural lan- guage parsing with spectral techniques and L- PCFGs, yielding a result that outperforms the orig- inal Berkeley parser from Petrov and Klein (2007). Instead …

Content truncated to 500 words