HENRY-CORE: Domain Adaptation and Stacking for Text Similarity

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 96–102, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics HENRY-CORE: Domain Adaptation and Stacking for Text Similarity∗ Michael Heilman and Nitin Madnani Educational Testing Service 660 Rosedale Road Princeton, NJ 08541, USA {mheilman,nmadnani}@ets.org Abstract This paper describes a system for automat- ically measuring the semantic similarity be- tween two texts, which was the aim of the 2013 Semantic Textual Similarity (STS) task (Agirre et al., 2013). For the 2012 STS task, Heilman and Madnani (2012) submitted the PERP system, which performed competitively in relation to other submissions. However, approaches including word and n-gram fea- tures also performed well (B¨ar et al., 2012; ˇSari´c et al., 2012), and the 2013 STS task fo- cused more on predicting similarity for text pairs from new domains. Therefore, for the three variations of our system that we were al- lowed to submit, we used stacking (Wolpert, 1992) to combine PERP with word and n- gram features and applied the domain adapta- tion approach outlined by Daume III (2007) to facilitate generalization to new domains. Our submissions performed well at most sub- tasks, particularly at measuring the similarity of news headlines, where one of our submis- sions ranked 2nd among 89 from 34 teams, but there is still room for improvement. 1 Introduction We aim to develop an automatic measure of the se- mantic similarity between two short texts (e.g., sen- tences). Such a measure could be useful for vari- ous applications, including automated short answer scoring (Leacock and Chodorow, 2003; Nielsen et al., 2008), question answering (Wang et al., 2007), ∗ System description papers for this task were required to have a team ID and task ID (e.g., “HENRY-CORE”) as a prefix. and machine translation evaluation (Przybocki et al., 2009). In this paper, we describe our submissions to the 2013 Semantic Textual Similarity (STS) task (Agirre et al., 2013), which evaluated implementations of text-to-text similarity measures. Submissions were evaluated according to Pearson correlations between gold standard similarity values acquired from hu- man raters and machine-produced similarity val- ues. Teams were allowed to submit up to three submissions. For each submission, correlations were calculated separately for four subtasks: mea- suring similarity between news headlines (“head- lines”), between machine translation outputs and hu- man reference translations (“SMT”), between word glosses from OntoNotes (Pradhan and Xue, 2009) and WordNet (Fellbaum, 1998) (“OnWN”), and be- tween frame descriptions from FrameNet (Fillmore et al., 2003) and glosses from WordNet (“FNWN”). A weighted mean of the correlations was also com- puted as an overall evaluation metric (the OnWn and FNWN datasets were smaller than the headlines and SMT datasets). The suggested training data for the 2013 STS task was the data from the 2012 STS task (Agirre et al., 2012), including both the training and test sets for that year. The 2012 task was similar ex- cept that the data were from a different set of sub- …

Content truncated to 500 words