Maximum Expected F-Measure Training of Logistic Regression Models

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 692–699, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Maximum Expected F-Measure Training of Logistic Regression Models Martin Jansche Center for Computational Learning Systems Columbia University New York, NY 10027, USA jansche@acm.org Abstract We consider the problem of training logis- tic regression models for binary classifi- cation in information extraction and infor- mation retrieval tasks. Fitting probabilis- tic models for use with such tasks should take into account the demands of the task- specific utility function, in this case the well-known F-measure, which combines recall and precision into a global measure of utility. We develop a training proce- dure based on empirical risk minimiza- tion / utility maximization and evaluate it on a simple extraction task. 1 Introduction Log-linear models have been used in many areas of Natural Language Processing (NLP) and Information Retrieval (IR). Scenarios in which log-linear models have been applied often involve simple binary clas- sification decisions or probability assignments, as in the following three examples: Ratnaparkhi et al. (1994) consider a restricted form of the prepositional phrase attachment problem where attachment deci- sions are binary; Ittycheriah et al. (2003) reduce en- tity mention tracking to the problem of modeling the probability of two mentions being linked; and Greiff and Ponte (2000) develop models of proba- bilistic information retrieval that involve binary de- cisions of relevance. What is common to all three approaches is the application of log-linear models to binary classification tasks.1 As Ratnaparkhi (1998, 1These kinds of log-linear models are also known among the NLP community as “maximum entropy models” (Berger et al., p. 27f.) points out, log-linear models of binary re- sponse variables are equivalent to, and in fact mere notational variants of, logistic regression models. In this paper we focus on binary classification tasks, and in particular on the loss or utility associ- ated with classification decisions. The three prob- lems mentioned before – prepositional phrase at- tachment, entity mention linkage, and relevance of a document to a query – differ in one crucial aspect: The first is evaluated in terms of accuracy or, equiva- lently, symmetric zero–one loss; but the second and third are treated as information extraction/retrieval problems and evaluated in terms of recall and preci- sion. Recall and precision are combined into a single overall utility function, the well-known F-measure. It may be desirable to estimate the parameters of a logistic regression model by maximizing F-measure during training. This is analogous, and in a cer- tain sense equivalent, to empirical risk minimiza- tion, which has been used successfully in related areas, such as speech recognition (Rahim and Lee, 1997), language modeling (Paciorek and Rosenfeld, 2000), and machine translation (Och, 2003). The novel contribution of this paper is a training procedure for (approximately) maximizing the ex- pected F-measure of a probabilistic classifier based on a logistic regression model. We formulate a vector-valued utility function which has a well- defined expected value; F-measure is then …

Content truncated to 500 words