Data Augmentation for Multiclass Utterance Classification – A Systematic Study

Proceedings of the 28th International Conference on Computational Linguistics, pages 5494–5506 Barcelona, Spain (Online), December 8-13, 2020 5494 Data Augmentation for Multiclass Utterance Classification – A Systematic Study Binxia Xu† Ant Financial Services Group† Max Planck Institute for Informatics‡ Hasso Plattner Institute, University of Potsdam§ Siyuan Qiu† Jie Zhang† Yafang Wang†∗ Xiaoyu Shen‡ Gerard de Melo§ Abstract Utterance classification is a key component in many conversational systems. However, classifying real-world user utterances is challenging, as people may express their ideas and thoughts in manifold ways, and the amount of training data for some categories may be fairly limited, resulting in imbalanced data distributions. To alleviate these is- sues, we conduct a comprehensive survey regarding data augmentation approaches for text classification, including simple random resampling, word-level transformations, and neural text generation to cope with imbalanced data. Our experiments focus on multi-class datasets with a large number of data samples, which has not been system- atically studied in previous work. The results show that the effectiveness of different data augmentation schemes depends on the nature of the dataset under consideration. 1 Introduction In modern conversational systems, classifying incoming user utterances is among the most crucial processes. This is particularly evident for automated customer service systems: if the underlying demand can successfully be classified based on the user’s description, a known solution can directly be provided to them. A weak classifier may miscategorize a request, resulting in customer dissatisfaction. A considerable cause for low-performing classification is a lack of sufficient training data for certain categories, which manifests as the problem of imbalanced data distributions. Classifying utterances is particularly challenging, as people may choose various different forms of expressing their ideas and thoughts. Thus, very dif- ferent utterances may reflect the same underlying intent. Recent findings on query variation in search systems suggest substantial potential for considering language diversity in NLP ap- plications (Koopman et al., 2017; Scells et al., 2018; Sultan et al., 2020). Motivated by these observations, we study how to improve utterance classification results by drawing on utter- ance variation. Unfortunately, soliciting clean human-generated data can be expensive and difficult to scale. In this study, we instead consider automated utterance generation schemes to augment the original dataset during the training process, in order to: (1) mitigate the data imbalance, and (2) improve the classification effectiveness. Since automatically generated data is cheap and easy to obtain, we can use it as a way of augmenting existing data, which may improve our classification model’s effectiveness and robustness. In this study, we conduct a thorough investigation of current data augmentation approaches to mitigate the imbalanced data problem, including simple resampling, word-level transfor- mations, and neural text generation. Subsequently, an optimal balanced spot is sought to achieve a better classification result. Contributions: The main contributions of this paper are twofold. First, we conduct a compre- hensive survey regarding state-of-the-art data augmentation approaches for multi-class text ∗ corresponding author, email: yafang.wyf@antfin.com This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/. …

Content truncated to 500 words