Improving the Quality of Minority Class Identification in Dialog Act Tagging

Proceedings of NAACL-HLT 2013, pages 802–807, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Improving the Quality of Minority Class Identification in Dialog Act Tagging Adinoyi Omuya 10gen New York, NY, USA wisdom@10gen.com Vinodkumar Prabhakaran CS, Columbia University New York, NY, USA vinod@cs.columbia.edu Owen Rambow CCLS, Columbia University New York, NY, USA rambow@ccls.columbia.edu Abstract We present a method of improving the perfor- mance of dialog act tagging in identifying mi- nority classes by using per-class feature opti- mization and a method of choosing the class based not on confidence, but on a cascade of classifiers. We show that it gives a minor- ity class F-measure error reduction of 22.8%, while also reducing the error for other classes and the overall error by about 10%. 1 Introduction In this paper, we discuss dialog act tagging, the task of assigning a dialog act to an utterance, where a dialog act (DA) is a high-level categorization of the pragmatic meaning of the utterance. Our data is email. Our starting point is the tagger described in (Hu et al., 2009), which uses a standard multi-class classifier based on support vector machines (SVMs). While the performance of this system is pretty good as measured by accuracy, it performs badly on the DA REQUEST-ACTION, which is a rare class. Multi- class SVMs are typically implemented as a set of SVMs, one per class, with the overall choice of class being determined by the SVM with the highest con- fidence (“one-against-all”). Multi-class SVMs are typically packaged as a single system, whose inner workings are ignored by the NLP researcher. In this paper we show that, for our problem of DA classi- fication, we can boost the performance of the rare classes (while maintaining the overall performance) by performing feature optimization separately for each individual classifier. But we also show that we can achieve an all-around error reduction by alter- ing the method by which the multi-class classifier combines the individual SVMs. This new method of combination is a simple cascade: we run the in- dividual classifiers in ascending order of frequency of the classes in the training corpus; the first classi- fier to classify the data point positively determines the choice of the overall classifier. If no classifier classifies the data point positively, we use the usual confidence-based method. This new method obtains a 22.8% error reduction for the minority class, and around 10% error reduction for the other classes and for the overall classifier. This paper is structured as follows. We start out by discussing related work (Section 2). We then present our data in Section 3, and in Section 4 we present the experiments with our systems and the re- sults. We report the results of an extrinsic evaluation in Section 5, and conclude. 2 Related Work Dialog act (DA) annotations and tagging, inspired by the speech act theory of Austin (1975) and Searle (1976), have been used in the NLP community to un- derstand and model dialog. Initial work was done on …

Content truncated to 500 words