ETS: Discriminative Edit Models for Paraphrase Scoring

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 529–535, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics ETS: Discriminative Edit Models for Paraphrase Scoring Michael Heilman and Nitin Madnani Educational Testing Service 660 Rosedale Road Princeton, NJ 08541, USA {mheilman,nmadnani}@ets.org Abstract Many problems in natural language process- ing can be viewed as variations of the task of measuring the semantic textual similarity be- tween short texts. However, many systems that address these tasks focus on a single task and may or may not generalize well. In this work, we extend an existing machine transla- tion metric, TERp (Snover et al., 2009a), by adding support for more detailed feature types and by implementing a discriminative learning algorithm. These additions facilitate applica- tions of our system, called PERP, to similar- ity tasks other than machine translation eval- uation, such as paraphrase recognition. In the SemEval 2012 Semantic Textual Similar- ity task, PERP performed competitively, par- ticularly at the two surprise subtasks revealed shortly before the submission deadline. 1 Introduction Techniques for measuring the similarity of two sen- tences have various potential applications: auto- mated short answer scoring (Nielsen et al., 2008; Leacock and Chodorow, 2003), question answering (Wang et al., 2007), machine translation evaluation (Przybocki et al., 2009; Snover et al., 2009a), etc. An important aspect of this problem is that sim- ilarity is not binary. Sentences can be very seman- tically similar, such that they might be called para- phrases of each other. They might be completely different. Or, they might be somewhere in between. Indeed, it is arguable that all sentence pairs (except exact duplicates) lie somewhere on a continuum of similarity. Therefore, it is desirable to develop meth- ods that model sentence pair similarity on a contin- uous, or at least ordinal, scale. In this paper, we describe a system for measuring the semantic similarity of pairs of short texts. As a starting point, we use the Translation Error Rate Plus (Snover et al., 2009a), or TERp, system, which was specifically developed for machine translation eval- uation. TERp takes two sentences as input, finds a set of weighted edits that convert one into the other with low overall weight, and then produces a length- normalized score. TERp also has a greedy, heuris- tic learning algorithm for inducing weights from la- beled sentence pairs in order to increase correlations with human similarity scores. Some features of the original TERp make adap- tation to other semantic similarity tasks difficult, in- cluding its largely one-to-one mapping of features to edits and its heuristic, greedy learning algorithm. For example, there is a single feature for lexical sub- stitution, even though it is clear that different types of substitutions have different effects on similarity (e.g., substituting “43.6” with “17” versus substitut- ing “a” for “an”). In addition, the heuristic learn- ing algorithm, which involves perturbing the weight vector by small amounts as in grid search, seems un- scalable to larger sets of overlapping features. Therefore, here, we use TERp’s inference …

Content truncated to 500 words