Increasing Maintainability of NLP Evaluation Modules Through Declarative Implementations

Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pages 1–2, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Increasing Maintainability of NLP Evaluation Modules Through Declarative Implementations Terry Heinze Research & Development Department Thomson Corporation Eagan, MN 55123 terry.heinze@thomson.com Marc Light Research & Development Department Thomson Corporation Eagan, MN 55123 marc.light@thomson.com Abstract Computing precision and recall metrics for named entity tagging and resolution involves classifying text spans as true positives, false positives, or false negatives. There are many factors that make this classification complicated for real world systems. We describe an evaluation system that attempts to control this complexity through a set of rules and a forward chaining inference engine. 1 Introduction Computing precision and recall metrics for named entity recognition systems involves classifying each text span that the system proposes as an entity and a subset of the text spans that the gold data specifies as an entity. These text spans must be classified as true positives, false posi- tives, or false negatives. In the simple case, it is easy to write a procedure to walk through the list of text spans from the system and check to see if a corresponding text span exists in the gold data with the same label, mark the text span as true posi- tive or false positive accordingly, and delete the span from the gold data set. Then the procedure need only walk through the remaining gold data set and mark these spans as false negatives. The three predicates are the equality of the span’s two offsets and the labels. This evaluation procedure is useful for any natural language processing task that involves finding and labeling text spans. The question this poster addresses is how best to man- age the complexity of the evaluation system that results from adding a number of additional requirements to the classification of text spans. The requirements may in- clude fuzzy extent predicates, label hierarchies, confi- dence levels for gold data, and collapsing multiple men- tions in a document to produce a single classification. In addition, named entity tasks often also involve resolving a mention of an entity to an entry in an authority file (i.e., record in a relational database). This extension also re- quires an interleaved evaluation where the error source is important. We started with a standard procedural approach, en- coding the logic in nested conditionals. When the nesting reached a depth of five (e.g., Figure 1), we decided to try another approach. We implemented the logic in a set of rules. More specifically, we used the Drools rules and for- ward chaining engine (http://labs.jboss.com/drools/) to classify text spans as true positives, false positives, and/or false negatives. The procedural code was 379 lines long. The declarative system consists of 25 rules with 150 lines of supporting code. We find the rules more modular and easier to modify and maintain. However, at this time, we have no experimental result to support this opinion. 2 Added Complexity of the Classification of Text …

Content truncated to 500 words