Nuggeteer: Automatic Nugget-Based Evaluation using Descriptions and Judgements

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 375–382, New York, June 2006. c⃝2006 Association for Computational Linguistics Nuggeteer: Automatic Nugget-Based Evaluation using Descriptions and Judgements Gregory Marton Alexey Radul Infolab Group, MIT CSAIL Cambridge, MA 02139 {gremio,axch}@mit.edu Abstract The TREC Definition and Relationship questions are evaluated on the basis of in- formation nuggets that may be contained in system responses. Human evalua- tors provide informal descriptions of each nugget, and judgements (assignments of nuggets to responses) for each response submitted by participants. While human evaluation is the most accurate way to compare systems, approximate automatic evaluation becomes critical during system development. We present Nuggeteer, a new automatic evaluation tool for nugget-based tasks. Like the first such tool, Pourpre, Nugge- teer uses words in common between can- didate answer and answer key to approx- imate human judgements. Unlike Pour- pre, but like human assessors, Nuggeteer creates a judgement for each candidate- nugget pair, and can use existing judge- ments instead of guessing. This cre- ates a more readily interpretable aggregate score, and allows developers to track in- dividual nuggets through the variants of their system. Nuggeteer is quantitatively comparable in performance to Pourpre, and provides qualitatively better feedback to developers. 1 Introduction The TREC Definition and Relationship questions are evaluated on the basis of information nuggets, abstract pieces of knowledge that, taken together, comprise an answer. Nuggets are described infor- mally, with abbreviations, misspellings, etc., and each is associated with an importance judgement: ‘vital’ or ‘okay’.1 In some sense, nuggets are like WordNet synsets, and their descriptions are like glosses. Responses may contain more than one nugget—when they contain more than one piece of knowledge from the answer. The median scores of today’s systems are frequently zero; most responses contain no nuggets (Voorhees, 2005). Human assessors decide what nuggets make up an answer based on some initial research and on pools of top system responses for each question. Answer keys list, for each nugget, its id, importance, and description; two example answer keys are shown in Figures 1 and 2. Assessors make binary deci- sions about each response, whether it contains each nugget. When multiple responses contain a nugget, the assessor gives credit only to the (subjectively) best response. Using the judgements of the assessors, the fi- nal score combines the recall of the available vi- tal nuggets, and the length (discounting whitespace) of the system response as a proxy for precision. Nuggets valued ‘okay’ contribute to precision by in- creasing the length allowance, but do not contribute to recall. The scoring formula is shown in Figure 3. 1Nuggeteer implements the pyramid scoring system from (Lin and Demner-Fushman, 2006), designed to soften the dis- 375 Qid 87.8: ’other’ question for target Enrico Fermi 1 vital belived in partical’s existence and named it neutrino 2 vital Called the atomic Bomb an evil thing 3 okay Achieved the first controlled nuclear chain reaction 4 vital Designed and built the first nuclear reactor 5 okay …

Content truncated to 500 words