CLaC-CORE: Exhaustive Feature Combination for Measuring Textual Similarity

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 202–206, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics CLaC-CORE: Exhaustive Feature Combination for Measuring Textual Similarity Ehsan Shareghi CLaC Laboratory Concordia University Montreal, QC H3G 1M8, CANADA eh share@cse.concordia.ca Sabine Bergler CLaC Laboratory Concordia University Montreal, QC H3G 1M8, CANADA bergler@cse.concordia.ca Abstract CLaC-CORE, an exhaustive feature combina- tion system ranked 4th among 34 teams in the Semantic Textual Similarity shared task STS 2013. Using a core set of 11 lexical features of the most basic kind, it uses a support vector regressor which uses a combination of these lexical features to train a model for predicting similarity between sentences in a two phase method, which in turn uses all combinations of the features in the feature space and trains separate models based on each combination. Then it creates a meta-feature space and trains a final model based on that. This two step pro- cess improves the results achieved by single- layer standard learning methodology over the same simple features. We analyze the correla- tion of feature combinations with the data sets over which they are effective. 1 Introduction The Semantic Textual Similarity (STS) shared task aims to find a unified way of measuring similarity between sentences. In fact, sentence similarity is a core element of tasks trying to establish how two pieces of text are related, such as Textual Entailment (RTE) (Dagan et al., 2006), and Paraphrase Recog- nition (Dolan et al., 2004). The STS shared task was introduced for SemEval-2012 and was selected as its first shared task. Similar in spirit, STS differs from the well-known RTE shared tasks in two important points: it defines a graded similarity scale to mea- sure similarity of two texts, instead of RTE’s binary yes/no decision and the similarity relation is consid- ered to be symmetrical, whereas the entailment rela- tion of RTE is inherently unidirectional. The leading systems in the 2012 competition used a variety of very simple lexical features. Each sys- tem combines a different set of related features. CLaC Labs investigated the different combination possibilities of these simple lexical features and measured their performance on the different data sets. Originally conceived to explore the space of all possible feature combinations for ‘feature com- bination selection’, a two-step method emerged that deliberately compiles and trains all feature combina- tions exhaustively and then trains an SVM regressor using all combination models as its input features. It turns out that this technique is not nearly as pro- hibitive as imagined and achieves statistically sig- nificant improvements over the alternative of feature selection or of using any one single combination in- dividually. We propose the method as a viable approach when the characteristics of the data are not well under- stood and no satisfactory training set is available. 2 Related Work Recently, systems started to approach measuring similarity by combining different resources and methods. For example, the STS-2012 shared task’s …

Content truncated to 500 words