User’s Choice of Precision and Recall in Named Entity Recognition

International Conference RANLP 2009 - Borovets, Bulgaria, pages 192–196 User’s Choice of Precision and Recall in Named Entity Recognition Roman Klinger and Christoph M. Friedrich Fraunhofer Institute for Algorithms and Scientific Computing (SCAI) Department of Bioinformatics Schloss Birlinghoven 53754 Sankt Augustin, Germany {roman.klinger,christoph.friedrich}@scai.fraunhofer.de Abstract Conditional Random Fields are commonly trained to maximize likelihood. The corresponding Fβ measure, the weighted harmonic mean of preci- sion and recall, which is established for evaluation in information retrieval and text mining, is not necessarily the optimal result for the user’s choice of β. Some approaches have been published to optimize multivariate measures like Fβ to overcome this inconsistency. The limitation is that constraints like the value of β have to be known at training time. This publication proposes a method of multi- objective optimization of both precision and recall based on a preceding likelihood training. The out- put is an estimation of pareto-optimal solutions from which the user can select the best for the actual application. Evaluated on two publicly available data sets in the field of named entity recognition, nearly all Fβ values are superior to those resulting from log-likelihood training. Keywords Named Entity Recognition, Conditional Random Fields, Multi- Objective Optimization, NSGA-II, Fβ measure, Recall, Precision 1 Introduction In information retrieval, the Fβ measure, the weighted harmonic mean between recall and precision, is estab- lished as evaluation measure. The corresponding β value to be chosen is application-depend. Methods for selecting β at training time exist for Support Vector Machines [18], Logistic Regression as well as Condi- tional Random Fields (CRF) [11] all of which are classi- cally optimized by means of accuracy-related measures [7, 8, 20]. A similar goal is known from the AmilCare system [4] with the main focus on user involvement. At inference time, a parameter to select between higher precision or recall can be introduced by chang- ing the decision threshold for an adequate decision function d(·) ∈ R. In sequential segmentation tasks like named entity recognition (NER), precision can be increased with this approach without retraining. In- creasing recall is possible with the allowance of overlaps as demonstrated for gene and protein names [3]. This requires the computation of reliable confidences, which increasing runtime is a drawback especially during in- ference [5, 20]. In contrast to optimizing one special value or select- ing the set of output entities in prediction phase, we propose to use an evolutionary optimization scheme to optimize recall and precision in a multi-objective way to yield different model configurations, which can be selected by an end-user depending on the respec- tive task with higher recall or higher precision without retraining. Thereby, the non-intentional choice of pre- cision and recall by optimization of accuracy (which is performed by maximizing the log-likelihood of the model given the training data in the case of CRFs) is avoided. The main contribution of this paper is therefore the presentation of multi-objective optimization for Con- ditional Random Fields (MOCRF). The feasibility of evolutionary optimization in such models is demon- strated. The …

Content truncated to 500 words