Single and multi-objective optimization for feature selection in anaphora resolution

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 93–101, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Single and Multi-objective Optimization for Feature Selection in Anaphora Resolution Sriparna Saha 1 Asif Ekbal 1 Olga Uryupina 2 Massimo Poesio 3,2 1 Department of Computer Science and Engineering, IIT Patna, India, {sriparna,asif}@iitp.ac.in 2 University of Trento, Center for Mind/Brain Sciences, uryupina@unitn.it 3 University of Essex, Language and Computation Group, poesio@essex.ac.uk Abstract There is no generally accepted met- ric for measuring the performance of anaphora resolution systems, and the ex- isting metrics—MUC, B3, CEAF, Blanc, among others—tend to reward signifi- cantly different behaviors. Systems op- timized according to one metric tend to perform poorly with respect to other ones, making it very difficult to compare anaphora resolution systems, as clearly shown by the results of the SEMEVAL 2010 Multilingual Coreference task. One so- lution would be to find a single com- pletely satisfactory metric, but it’s not clear whether this is possible and at any rate it is not going to happen any time soon. An alternative is to optimize mod- els according to multiple metrics simulta- neously. In this paper, we show, first of all, that this is possible to develop such models using Multi-objective Optimiza- tion (MOO) techniques based on Genetic Algorithms. Secondly, we show that op- timizing according to multiple metrics si- multaneously may result in better results with respect to each individual metric than optimizing according to that metric only. 1 Introduction In anaphora resolution,1 as in other HLT tasks, optimization to a metric is essential to achieve good performance (Hoste, 2005; Uryupina, 2010). However, many evaluation metrics have been pro- posed for anaphora resolution, each capturing what seems to be a key intuition about the task: from MUC (Vilain et al., 1995) to B3 (Bagga and 1We use the term ’anaphora resolution’ to refer to the task perhaps most commonly referred to as ’coreference resolu- tion,’ which many including us find a misnomer. For the pur- poses of the present paper the two terms could be seen as interchangeable. Baldwin, 1998), from the ACE metric (Doddington et al., 2004) to CEAF (Luo, 2005) to BLANC (Re- casens and Hovy, 2011). And unlike in other areas of HLT, none has really taken over. This would not matter so much if those metrics were to reward the same systems; but in fact, as dramatically demon- strated by the results of the Coreference Task at SEMEVAL 2010 (Recasens et al., 2010), the oppo- site is true—almost every system could come on top depending on which metric was chosen. It seems unlikely that the field will converge on a single metric any time soon. Given that many of the proposed metrics do capture what would seem to be plausible intuitions, it would seem desirable to develop methods to optimize systems according to more than one metric at once—in particular, ac- cording to at least one metric of what we might call the ’link-based’ cluster of metrics …

Content truncated to 500 words