The SemEval-2007 WePS Evaluation: Establishing a benchmark for the Web People Search Task

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 64–69, Prague, June 2007. c⃝2007 Association for Computational Linguistics The SemEval-2007 WePS Evaluation: Establishing a benchmark for the Web People Search Task Javier Artiles UNED NLP & IR group Madrid, Spain javart@bec.uned.es nlp.uned.es/∼javier Julio Gonzalo UNED NLP & IR group Madrid, Spain julio@lsi.uned.es nlp.uned.es/∼julio Satoshi Sekine Computer Science Department New York University, USA sekine@cs.nyu.edu nlp.cs.nyu.edu/sekine Abstract This paper presents the task definition, re- sources, participation, and comparative re- sults for the Web People Search task, which was organized as part of the SemEval-2007 evaluation exercise. This task consists of clustering a set of documents that mention an ambiguous person name according to the actual entities referred to using that name. 1 Introduction Finding information about people in the World Wide Web is one of the most common activities of Internet users. Person names, however, are highly ambigu- ous. In most cases, the results for a person name search are a mix of pages about different people sharing the same name. The user is then forced ei- ther to add terms to the query (probably losing recall and focusing on one single aspect of the person), or to browse every document in order to filter the infor- mation about the person he is actually looking for. In an ideal system the user would simply type a person name, and receive search results clustered ac- cording to the different people sharing that name. And this is, in essence, the WePS (Web People Search) task we have proposed to SemEval-2007 participants: systems receive a set of web pages (which are the result of a web search for a per- son name), and they have to cluster them in as many sets as entities sharing the name. This task has close links with Word Sense Disambiguation (WSD), which is generally formulated as the task of deciding which sense a word has in a given con- text. In both cases, the problem addressed is the res- olution of the ambiguity in a natural language ex- pression. A couple of differences make our prob- lem different. WSD is usually focused on open- class words (common nouns, adjectives, verbs and adverbs). The first difference is that boundaries be- tween word senses in a dictionary are often subtle or even conflicting, making binary decisions harder and sometimes even useless depending on the ap- plication. In contrast, distinctions between people should be easier to establish. The second difference is that WSD usually operates with a dictionary con- taining a relatively small number of senses that can be assigned to each word. Our task is rather a case of Word Sense Discrimination, because the number of “senses” (actual people) is unknown a priori, and it is in average much higher than in the WSD task (there are 90,000 different names shared by 100 mil- lion people according to the U.S. Census Bureau). There is also a strong relation of our proposed task with the Co-reference Resolution problem, fo- cused …

Content truncated to 500 words