CECL: a New Baseline and a Non-Compositional Approach for the Sick Benchmark

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 160–165, Dublin, Ireland, August 23-24, 2014. CECL: a New Baseline and a Non-Compositional Approach for the Sick Benchmark Yves Bestgen Centre for English Corpus Linguistics Universit´e catholique de Louvain yves.bestgen@uclouvain.be Abstract This paper describes the two procedures for determining the semantic similarities between sentences submitted for the Se- mEval 2014 Task 1. MeanMaxSim, an unsupervised procedure, is proposed as a new baseline to assess the efficiency gain provided by compositional models. It out- performs a number of other baselines by a wide margin. Compared to the word- overlap baseline, it has the advantage of taking into account the distributional simi- larity between words that are also involved in compositional models. The second procedure aims at building a predictive model using as predictors MeanMaxSim and (transformed) lexical features describ- ing the differences between each sentence of a pair. It finished sixth out of 17 teams in the textual similarity sub-task and sixth out of 19 in the textual entailment sub- task. 1 Introduction The SemEval-2014 Task 1 (Marelli et al., 2014a) was designed to allow a rigorous evaluation of compositional distributional semantic models (CDSMs). CDSMs aim to represent the meaning of phrases and sentences by composing the dis- tributional representations of the words they con- tain (Baroni et al., 2013; Bestgen and Cabiaux, 2002; Erk and Pado, 2008; Grefenstette, 2013; Kintsch, 2001; Mitchell and Lapata, 2010); they are thus an extension of Distributional Semantic Models (DSMs), which approximate the meaning of words with vectors summarizing their patterns of co-occurrence in a corpus (Baroni and Lenci, This work is licenced under a Creative Commons Attribution 4.0 International License. Page numbers and proceedings footer are added by the organizers. License details: http: //creativecommons.org/licenses/by/4.0/ 2010; Bestgen et al., 2006; Kintsch, 1998; Lan- dauer and Dumais, 1997). The dataset for this task, called SICK (Sentences Involving Composi- tional Knowledge), consists of almost 10,000 En- glish sentence pairs annotated for relatedness in meaning and entailment relation by ten annotators (Marelli et al., 2014b). The rationale behind this dataset is that ”un- derstanding when two sentences have close mean- ings or entail each other crucially requires a com- positional semantics step” (Marelli et al., 2014b), and thus that annotators judge the similarity be- tween the two sentences of a pair by first build- ing a mental representation of the meaning of each sentence and then comparing these two represen- tations. However, another option was available to the annotators. They could have paid atten- tion only to the differences between the sentences, and assessed the significance of these differences. Such an approach could have been favored by the dataset built on the basis of a thousand sentences modified by a limited number of (often) very specific transformations, producing sentence pairs that might seem quite repetitive. An analysis con- ducted during the training phase of the challenge brought some support for this hypothesis. The analysis focused on pairs of sentences in which the only difference between the …

Content truncated to 500 words