Uptraining for Accurate Deterministic Question Parsing

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 705–713, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Uptraining for Accurate Deterministic Question Parsing Slav Petrov, Pi-Chuan Chang, Michael Ringgaard, Hiyan Alshawi Google Research {slav,pichuan,ringgaard,hiyan}@google.com Abstract It is well known that parsing accuracies drop significantly on out-of-domain data. What is less known is that some parsers suffer more from domain shifts than others. We show that dependency parsers have more difficulty parsing questions than constituency parsers. In particular, deterministic shift-reduce depen- dency parsers, which are of highest interest for practical applications because of their lin- ear running time, drop to 60% labeled accu- racy on a question test set. We propose an uptraining procedure in which a deterministic parser is trained on the output of a more ac- curate, but slower, latent variable constituency parser (converted to dependencies). Uptrain- ing with 100K unlabeled questions achieves results comparable to having 2K labeled ques- tions for training. With 100K unlabeled and 2K labeled questions, uptraining is able to improve parsing accuracy to 84%, closing the gap between in-domain and out-of-domain performance. 1 Introduction Parsing accuracies on the popular Section 23 of the Wall Street Journal (WSJ) portion of the Penn Tree- bank have been steadily improving over the past decade. At this point, we have many different pars- ing models that reach and even surpass 90% depen- dency or constituency accuracy on this test set (Mc- Donald et al., 2006; Nivre et al., 2007; Charniak and Johnson, 2005; Petrov et al., 2006; Carreras et al., 2008; Koo and Collins, 2010). Quite impressively, models based on deterministic shift-reduce parsing algorithms are able to rival the other computation- ally more expensive models (see Nivre (2008) and references therein for more details). Their linear running time makes them ideal candidates for large scale text processing, and our model of choice for this paper. Unfortunately, the parsing accuracies of all mod- els have been reported to drop significantly on out- of-domain test sets, due to shifts in vocabulary and grammar usage (Gildea, 2001; McClosky et al., 2006b; Foster, 2010). In this paper, we focus our attention on the task of parsing questions. Questions pose interesting challenges for WSJ-trained parsers because they are heavily underrepresented in the training data (there are only 334 questions among the 39,832 training sentences). At the same time, questions are of particular interest for user facing applications like question answering or web search, which necessitate parsers that can process questions in a fast and accurate manner. We start our investigation in Section 3 by train- ing several state-of-the-art (dependency and con- stituency) parsers on the standard WSJ training set. When evaluated on a question corpus, we observe dramatic accuracy drops exceeding 20% for the de- terministic shift-reduce parsers. In general, depen- dency parsers (McDonald et al., 2006; Nivre et al., 2007), seem to suffer more from this domain change than constituency parsers (Charniak and Johnson, 2005; Petrov et al., 2006). Overall, the latent vari- able …

Content truncated to 500 words