Evaluation Metrics for the Lexical Substitution Task

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 289–292, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Evaluation Metrics for the Lexical Substitution Task Sanaz Jabbari Mark Hepple Louise Guthrie Department of Computer Science, University of Sheffield 211 Portobello Street, Sheffield, S1 4DP, UK {S.Jabbari,M.Hepple,L.Guthrie}@dcs.shef.ac.uk Abstract We identify some problems of the evaluation metrics used for the English Lexical Substitu- tion Task of SemEval-2007, and propose al- ternative metrics that avoid these problems, which we hope will better guide the future de- velopment of lexical substitution systems. 1 Introduction The English Lexical Substitution task at SemEval- 2007 (here called ELS07) requires systems to find substitutes for target words in a given sentence (Mc- Carthy & Navigli, 2007: M&N). For example, we might replace the target word match with game in the sentence they lost the match. System outputs are evaluated against a set of candidate substitutes pro- posed by human subjects for test items. Targets are typically sense ambiguous (e.g. match in the above example), and so task performance requires a com- bination of word sense disambiguation (by exploit- ing the given sentential context) and (near) synonym generation. In this paper, we discuss some problems of the evaluation metrics used in ELS07, and then propose some alternative measures that avoid these problems, and which we believe will better serve to guide the development of lexical substitution sys- tems in future work.1 The subtasks within ELS07 divide into two groups, in terms of whether they fo- cus on a system’s ‘best’ answer for a test item, or ad- dress the broader set of answer candidates a system 1We consider here only the case of substituting for single word targets. Subtasks of ELS07 involving multi-word substi- tutions are not addressed. can produce. In what follows, we address these two cases in separate sections, and then present some re- sults for applying our new metrics for the second case. We begin by briefly introducing the test ma- terials that were created for the ELS07 evaluation. 2 Evaluation Materials Briefly stated, the ELS07 dataset comprises around 2000 sentences, providing 10 test sentences each for some 201 preselected target words, which were required to be sense ambiguous and have at least one synonym, and which include nouns, verbs, ad- jectives and adverbs. Five human annotators were asked to suggest up to three substitutes for the tar- get word of each test sentence, and their collected suggestions serve as the gold standard against which system outputs are compared. Around 300 sentences were distributed as development data, and the re- mainder retained for the final evaluation. To assist defining our metrics, we formally de- scribe this data as follows.2 For each sentence ti in the test data (1 ≤ i ≤ N, N the number of test items), let Hi denote the set of human proposed sub- stitutes. A key aspect of the data is the count of hu- man annotators that proposed each candidate (since a …

Content truncated to 500 words