Investigating Loss Functions and Optimization Methods for Discriminative Learning of Label Sequences

Investigating Loss Functions and Optimization Methods for Discriminative Learning of Label Sequences Yasemin Altun Computer Science Brown University Providence, RI 02912 altun@cs.brown.edu Mark Johnson Cognitive and Linguistic Sciences Brown University Providence, RI 02912 Mark Johnson@brown.edu Thomas Hofmann Computer Science Brown University Providence, RI 02912 th@cs.brown.edu Abstract Discriminative models have been of inter- est in the NLP community in recent years. Previous research has shown that they are advantageous over generative mod- els. In this paper, we investigate how dif- ferent objective functions and optimiza- tion methods affect the performance of the classifiers in the discriminative learning framework. We focus on the sequence la- belling problem, particularly POS tagging and NER tasks. Our experiments show that changing the objective function is not as effective as changing the features in- cluded in the model. 1 Introduction Until recent years, generative models were the most common approach for many NLP tasks. Recently, there is a growing interest on discriminative mod- els in the NLP community, and these models were shown to be successful for different tasks(Lafferty et al., 2001; Ratnaparkhi, 1999; Collins, 2000). Dis- criminative models do not only have theoretical ad- vantages over generative models, as we discuss in Section 2, but they are also shown to be empirically favorable over generative models when features and objective functions are fixed (Klein and Manning, 2002). In this paper, we use discriminative models to investigate the optimization of different objective functions by a variety of optimization methods. We focus on label sequence learning tasks. Part-of- Speech (POS) tagging and Named Entity Recogni- tion (NER) are the most studied applications among these tasks. However, there are many others, such as chunking, pitch accent prediction and speech edit detection. These tasks differ in many aspects, such as the nature of the label sequences (chunks or indi- vidual labels), their difficulty and evaluation meth- ods. Given this variety, we think it is worthwhile to investigate how optimizing different objective func- tions affects performance. In this paper, we varied the scale (exponential vs logarithmic) and the man- ner of the optimization (sequential vs pointwise) and using different combinations, we designed 4 differ- ent objective functions. We optimized these func- tions on NER and POS tagging tasks. Despite our intuitions, our experiments show that optimizing ob- jective functions that vary in scale and manner do not affect accuracy much. Instead, the selection of the features has a larger impact. The choice of the optimization method is impor- tant for many learning problems. We would like to use optimization methods that can handle a large number of features, converge fast and return sparse classifiers. The importance of the features, and therefore the importance of the ability to cope with a larger number of features is well-known. Since training discriminative models over large corpora can be expensive, an optimization method that con- verges fast might be advantageous over others. A sparse classifier has a shorter test time than a denser classifier. For applications in which the test time is crucial, optimization …

Content truncated to 500 words