Balancing via Generation for Multi-Class Text Classification Improvement

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1440–1452 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1440 Balancing via Generation for Multi-Class Text Classification Improvement Naama Tepper ∗, Esther Goldbraich ∗, Naama Zwerdling, George Kour, Ateret Anaby-Tavor, Boaz Carmeli IBM Research {naama.tepper, esthergold, naamaz, atereta, boazc}@il.ibm.com, gkour@ibm.com Abstract Data balancing is a known technique for im- proving the performance of classification tasks. In this work we define a novel balancing-via- generation framework termed BalaGen. Bala- Gen consists of a flexible balancing policy cou- pled with a text generation mechanism. Com- bined, these two techniques can be used to aug- ment a dataset for more balanced distribution. We evaluate BalaGen on three publicly avail- able semantic utterance classification (SUC) datasets. One of these is a new COVID-19 Q&A dataset published here for the first time. Our work demonstrates that optimal balanc- ing policies can significantly improve classi- fier performance, while augmenting just part of the classes and under-sampling others. Fur- thermore, capitalizing on the advantages of balancing, we show its usefulness in all rele- vant BalaGen framework components. We val- idate the superiority of BalaGen on ten seman- tic utterance datasets taken from real-life goal- oriented dialogue systems. Based on our re- sults we encourage using data balancing prior to training for text classification tasks. 1 Introduction Imbalanced datasets pose a known difficulty in achieving ultimate classification performance as classifiers tend to be biassed towards larger classes (Guo et al., 2008; Japkowicz and Stephen, 2002; Japkowicz, 2000). Moreover, identifying samples that belong to under-represented classes is of high importance in many real-life domains such as fraud detection, disease diagnosis, and cyber security. Although the imbalanced data classification problem is well-defined, and has been researched extensively over the last two decades (Estabrooks et al., 2004; Batista et al., 2004; Ramyachitra and Manikandan, 2014; Zhu et al., 2017; Buda et al., ∗Equal contribution 2018), there has been considerably less work de- voted to balancing textual datasets. We propose a novel balancing-via-generation framework, termed BalaGen, to improves textual classification performance. BalaGen uses a balanc- ing policy to identify over- and under-represented classes. It then uses controlled text generation, coupled with a weak labeling mechanism to aug- ment the under-represented classes. Additionally, it applies under-sampling to decrease the over- represented classes. Our analysis is focused on semantic utterance classification (SUC) (Tur et al., 2012; Tur and Deng, 2011; Schuurmans and Frasincar, 2019). SUC is a fundamental, multi-class, highly imbal- anced textual classification problem. For example, it is widely used for intent (class) detection in goal- oriented dialogue systems (Henderson et al., 2014; Bohus and Rudnicky, 2009), and for frequently asked question (FAQ) retrieval (Sakata et al., 2019; Gupta and Carvalho, 2019; Wang et al., 2017). Correctly identifying scarce utterances is of great importance in many real life scenarios. For exam- ple, consider a scenario in which a user converses with the dialogue system in an online shop (Yan et al., 2017). For the store owner, the task of cor- rectly …

Content truncated to 500 words