Category Mapping for Zero-shot Text Classification

Category Mapping for Zero-shot Text Classification Qiu-Xia Zhang∗ Department of Computer Science and Information Engineering National Taiwan University r10922164@ntu.edu.tw Te-Yu Chi∗ Department of Computer Science and Information Engineering National Taiwan University d09922009@ntu.edu.tw Te-Lun Yang∗ Department of Computer Science and Information Engineering National Taiwan University d12944007@ntu.edu.tw Yu-Meng Tang∗ Department of Computer Science and Information Engineering Tongji University tonmoregulus@gmail.com Ta-Lin Chen∗ Department of Electrical and Computer Engineering University of Texas at Austin talin@utexas.edu Jyh-Shing Roger Jang Department of Computer Science and Information Engineering National Taiwan University jang@mirlab.org Abstract The existing method of using large pre-trained models with prompts for zero-shot text classifi- cation possesses powerful representation abil- ity and scalability. However, its commercial availability is relatively limited. The approach of employing class labels and existing datasets to fine-tune smaller models for zero-shot classi- fication is comparatively straightforward, yet it might lead to weaker model generalization ability. This paper introduces three meth- ods to enhance the accuracy and generaliza- tion capability of pre-trained models in zero- shot text classification tasks: 1) utilizing pre- trained language models and structuring in- puts into a standardized multiple-choice for- mat; 2) creating a text classification training dataset using Wikipedia text data and refin- ∗These authors contributed equally to this work. ing the pre-trained model through fine-tuning; and 3) suggesting a zero-shot category map- ping technique based on GloVe text similar- ity, wherein Wikipedia categories replace tex- tual categories. Remarkably, without employ- ing labeled samples for fine-tuning, the pro- posed method achieves results comparable to the best models fine-tuned with labeled sam- ples. Natural Language Processing, Pre-trained Language Models, Zero-shot Text Classifica- tion, Classification, GloVe 1 Introduction Text classification is a pivotal task within the realm of natural language processing, with ex- tensive applications in areas such as spam fil- tering, information retrieval, personalized rec- ommendations, sentiment analysis, and pub- 1 The 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023) Taipei City, Taiwan, October 20-21, 2023. The Association for Computational Linguistics and Chinese Language Processing lic opinion monitoring. Presently, pre-trained models, after fine-tuning on labeled data, have achieved substantial accuracy improvements on these labeled datasets. However, there are inherent limitations when relying solely on supervised methods in practical applications. One primary concern is the necessity to con- struct a new dataset for each novel new task, involving significant data collection and man- ual annotation efforts. This process conse- quently escalates both time and labor costs. Particularly containing multiple tasks of text classification, incomplete data collection may lead to issues associated with data sparsity. The zero-shot classification model stands out due to its cross-domain universality, no need for manual labeling for new tasks, thus considerably saving time and lobar costs. Two primary approaches are currently prevalent for zero-shot classification. One employs the prompt (Brown et al., 2020) method, lever- aging the contextual attention mechanism of large pre-trained language models, using prompts to guide the model to generate the desired responses. However, the leverage of these large models often depends on commer- cial APIs like GPT and ChatGPT, …

Content truncated to 500 words