Application of Mix-Up Method in Document Classification Task Using BERT

Proceedings of Recent Advances in Natural Language Processing, pages 679–683 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_077 679 Application of Mix-Up Method in Document Classification Task Using BERT Naoki Kikuta and Hiroyuki Shinnou Faculty of Engineering, Ibaraki University, Japan {21nm720n, hiroyuki.shinnou.0828}@vc.ibaraki.ac.jp Abstract The mix-up method (Zhang et al., 2017), one of the methods for data augmentation, is known to be easy to implement and highly ef- fective. Although the mix-up method is in- tended for image identification, it can also be applied to natural language processing. In this paper, we attempt to apply the mix-up method to a document classification task using bidirec- tional encoder representations from transform- ers (BERT) (Devlin et al., 2018). Since BERT allows for two-sentence input, we concate- nated word sequences from two documents with different labels and used the multi-class output as the supervised data with a one-hot vector. In an experiment using the livedoor news corpus, which is Japanese, we compared the accuracy of document classification using two methods for selecting documents to be concatenated with that of ordinary document classification. As a result, we found that the proposed method is better than the normal classification when the documents with labels shortages are mixed preferentially. This indi- cates that how to choose documents for mix-up has a significant impact on the results. 1 Introduction The high cost of constructing training data is al- ways a problem when solving natural language processing (NLP) tasks using machine learning approaches. Several attempts have been made to solve this problem. One of the most recent methods for constructing training data is data augmentation (Shorten and Khoshgoftaar, 2019). Data augmen- tation methods can be divided into two types: pro- cessing and generation. For image identification, even if an image in the training data is flipped or cropped, the image label does not change. This means that the training data can be increased by adding such processed images to the training data. Alternatively, the method for generating artificial data using generative adversarial network can be considered as a type of data augmentation. The mix-up method is one of the methods for generat- ing data augmentation. It is highly effective and can easily be implemented. Although the mix-up method is used for image identification, it can also be useed for NLP. 2 Related topic 2.1 Bidirectional encoder representations from transformers (BERT) BERT is a high-performance, pre-trained model that has been widely used since its creation by Google (2018) (Devlin et al., 2018). It can be used for classification, word prediction and context de- termination. In this study, we improve the accuracy of the BERT-based document classification task using the mix-up method. 2.2 Mix-up method Mix-up is a data augmentation method in the field of image proposed by Hongyi Zhang (2017) (Zhang et al., 2017). The data augmentation is performed using Equations 1 and 2 for image data and labels respectively. x = λxi + (1 − λ)xj (1) y = λyi + (1 − λ)yj (2) x is a vector of image data , …

Content truncated to 500 words