Toward a Task-based Gold Standard for Evaluation of NP Chunks and Technical Terms

Toward a Task-based Gold Standard for Evaluation of NP Chunks and Technical Terms Nina Wacholder Rutgers University nina@scils.rutgers.edu Peng Song Rutgers University psong@paul.rutgers.edu Abstract We propose a gold standard for evaluating two types of information extraction output -- noun phrase (NP) chunks (Abney 1991; Ramshaw and Marcus 1995) and technical terms (Justeson and Katz 1995; Daille 2000; Jacquemin 2002). The gold standard is built around the notion that since different semantic and syntactic variants of terms are arguably correct, a fully satisfactory assess- ment of the quality of the output must include task-based evaluation. We conducted an experi- ment that assessed subjects’ choice of index terms in an information access task. Subjects showed significant preference for index terms that are longer, as measured by number of words, and more complex, as measured by number of prepo- sitions. These terms, which were identified by a human indexer, serve as the gold standard. The experimental protocol is a reliable and rigorous method for evaluating the quality of a set of terms. An important advantage of this task-based evalua- tion is that a set of index terms which is different than the gold standard can ‘win’ by providing better information access than the gold standard itself does. And although the individual human subject experiments are time consuming, the ex- perimental interface, test materials and data analysis programs are completely re-usable. 1 Introduction The standard metrics for evaluation of the output of NLP systems are precision and recall. Given an ar- guably correct list of the units that a system would identify if it performed perfectly, there should in principle be no discrepancy between the units identi- fied by a system and the units that are either useful in a particular application or are preferred by human beings for use in a particular task. But when the satis- factory output can take many different forms, as in summarization and generation, evaluation by preci- sion and recall is not sufficient. In these cases, the challenge for system designers and users is to effec- tively distinguish between systems that provide gen- erally satisfactory output and systems that do not. NP chunks (Abney 1991; Ramshaw and Marcus 1995; Evans and Zhai 1996; Frantzi and Ananiadou 1996) and technical terms (Dagan and Church 1994; Justeson and Katz 1995; Daille 1996; Jacquemin 2001; Bourigault et al. 2002) fall into this difficult-to- assess category. NPs are recursive structures. For the maximal NP large number of recent newspaper articles on biomedical science and clinical practice, a full- fledged parser would legitimately identify (at least) seven NPs in addition to the maximal one: large number; recent newspaper articles; large number of recent newspaper articles; biomedical science; clini- cal practice; biomedical science and clinical prac- tice; and recent newspaper articles on biomedical science and clinical practice. To evaluate the per- formance of a parser, NP chunks can usefully be evaluated by a gold standard; many systems (e.g., Ramshaw and Marcus 1995 and Cardie and Pierce 1988) use the Penn Treebank for this type …

Content truncated to 500 words