That’s Not What I Meant! Using Parsers to Avoid Structural Ambiguities in Generated Text

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 413–423, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics That’s Not What I Meant! Using Parsers to Avoid Structural Ambiguities in Generated Text Manjuan Duan and Michael White Department of Linguistics The Ohio State University Columbus, OH 43210, USA {duan,mwhite}@ling.osu.edu Abstract We investigate whether parsers can be used for self-monitoring in surface real- ization in order to avoid egregious errors involving “vicious” ambiguities, namely those where the intended interpretation fails to be considerably more likely than alternative ones. Using parse accuracy in a simple reranking strategy for self- monitoring, we find that with a state- of-the-art averaged perceptron realization ranking model, BLEU scores cannot be improved with any of the well-known Treebank parsers we tested, since these parsers too often make errors that human readers would be unlikely to make. How- ever, by using an SVM ranker to combine the realizer’s model score together with features from multiple parsers, including ones designed to make the ranker more ro- bust to parsing mistakes, we show that sig- nificant increases in BLEU scores can be achieved. Moreover, via a targeted man- ual analysis, we demonstrate that the SVM reranker frequently manages to avoid vi- cious ambiguities, while its ranking errors tend to affect fluency much more often than adequacy. 1 Introduction Rajkumar & White (2011; 2012) have recently shown that some rather egregious surface realiza- tion errors—in the sense that the reader would likely end up with the wrong interpretation—can be avoided by making use of features inspired by psycholinguistics research together with an other- wise state-of-the-art averaged perceptron realiza- tion ranking model (White and Rajkumar, 2009), as reviewed in the next section. However, one is apt to wonder: could one use a parser to check whether the intended interpretation is easy to re- cover, either as an alternative or to catch additional mistakes? Doing so would be tantamount to self- monitoring in Levelt’s (1989) model of language production. Neumann & van Noord (1992) pursued the idea of self-monitoring for generation in early work with reversible grammars. As Neumann & van Noord observed, a simple, brute-force way to gen- erate unambiguous sentences is to enumerate pos- sible realizations of an input logical form, then to parse each realization to see how many inter- pretations it has, keeping only those that have a single reading; they then went on to devise a more efficient method of using self-monitoring to avoid generating ambiguous sentences, targeted to the ambiguous portion of the output. We might question, however, whether it is really possible to avoid ambiguity entirely in the general case, since Abney (1996) and others have argued that nearly every sentence is potentially ambiguous, though we (as human comprehenders) may not notice the ambiguities if they are unlikely. Tak- ing up this issue, Khan et al. (2008)—building on Chantree et al.’s (2006) approach to identifying “innocuous” ambiguities—conducted several ex- periments to test whether ambiguity could be bal- anced against …

Content truncated to 500 words