Lexical Reference: a Semantic Matching Subtask

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 172–179, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Lexical Reference: a Semantic Matching Subtask Oren Glickman and Eyal Shnarch and Ido Dagan Computer Science Department Bar Ilan University Ramat Gan, Israel {glikmao, dagan}@cs.biu.ac.il Abstract Semantic lexical matching is a prominent subtask within text understanding applica- tions. Yet, it is rarely evaluated in a di- rect manner. This paper proposes a def- inition for lexical reference which cap- tures the common goals of lexical match- ing. Based on this definition we created and analyzed a test dataset that was uti- lized to directly evaluate, compare and im- prove lexical matching models. We sug- gest that such decomposition of the global semantic matching task is critical in order to fully understand and improve individual components. 1 Introduction A fundamental task for text understanding ap- plications is to identify semantically equivalent pieces of text. For example, Question Answer- ing (QA) systems need to match corresponding parts in the question and in the answer passage, even though such parts may be expressed in dif- ferent terms. Summarization systems need to rec- ognize (redundant) semantically matching parts in multiple sentences that are phrased differently. Other applications, such as information extraction and retrieval, face pretty much the same seman- tic matching task. The degree of semantic match- ing found is typically factored into systems’ scor- ing and ranking mechanisms. The recently pro- posed framework of textual entailment (Dagan et al., 2006) attempts to formulate the generic seman- tic matching problem in an application indepen- dent manner. The most commonly implemented semantic matching component addresses the lexical level. At this level the goal is to identify whether the meaning of a lexical item of one text is expressed also within the other text. Typically, lexical match- ing models measure the degree of literal lexical overlap, augmented with lexical substitution cri- teria based on resources such as Wordnet or the output of statistical similarity methods (see Sec- tion 2). Many systems apply semantic matching only at the lexical level, which is used to approx- imate the overall degree of semantic matching be- tween texts. Other systems incorporate lexical matching as a component within more complex models that examine matching at higher syntactic and semantic levels. While lexical matching models are so promi- nent within semantic systems they are rarely eval- uated in a direct manner. Typically, improve- ments to a lexical matching model are evaluated by their marginal contribution to overall system per- formance. Yet, such global and indirect evaluation does not indicate the absolute performance of the model relative to the sheer lexical matching task for which it was designed. Furthermore, the indi- rect application-dependent evaluation mode does not facilitate improving lexical matching models in an application dependent manner, and does not allow proper comparison of such models which were developed (and evaluated) by different re- searchers within different systems. This paper proposes a generic definition for the lexical matching task, …

Content truncated to 500 words