Ranking-Constrained Learning with Rationales for Text Classification

Findings of the Association for Computational Linguistics: ACL 2022, pages 2034 - 2046 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Ranking-Constrained Learning with Rationales for Text Classification Juanyan Wang Illinois Institute of Technology Chicago, IL USA jwang245@hawk.iit.edu Manali Sharma Samsung Semiconductor, Inc San Jose, CA USA manali.s@samsung.com Mustafa Bilgic Illinois Institute of Technology Chicago, IL USA mbilgic@iit.edu Abstract We propose a novel approach that jointly uti- lizes the labels and elicited rationales for text classification to speed up the training of deep learning models with limited training data. We define and optimize a ranking-constrained loss function that combines cross-entropy loss with ranking losses as rationale constraints. We evaluate our proposed rationale-augmented learning approach on three human-annotated datasets, and show that our approach pro- vides significant improvements over classifi- cation approaches that do not utilize ratio- nales as well as other state-of-the-art rationale- augmented baselines. 1 Introduction Text classification has been used for numerous ap- plications including sentiment analysis (Hemma- tian and Sohrabi, 2019), information retrieval (Ag- garwal and Zhai, 2012), and language identification (Jauhiainen et al., 2019). When presented with a large number of labeled documents, common text classification models demonstrate impressive re- sults. In practical settings, however, labeled data is often scarce. Labeling documents is a tedious task that requires time and effort, thus curating a large labeled corpus can be expensive and even unrealistic. There is a wide range of use cases for businesses and industry that require curating a labeled dataset for the current task before the need to move on to the next task arises. For example, consider le- gal case document classification where documents need to be labeled as relevant/not-relevant to the current case at hand. The next legal case requires labeling the documents as relevant/not-relevant for that particular case, and so on. Similarly, several fast-response tasks such as immediate analysis of news and social media posts for a breaking news, for a recently released product, for a policy an- nouncement, etc., require fast curation of a small and yet informative labeled dataset. Label: negative Label: positive I do not find this show at all funny. I actually think it is much worse than any of the other terrible Disney channel sit-coms right now. I love this movie and have seen it quite a few times over the years. It does get better with every viewing. I agree with all of the positive reviews here. Figure 1: Rationales annotated on a negative movie re- view and a positive movie review. An effective approach to make the best use of the human’s time and maximize classifier performance with a small labeled dataset is to elicit rich feed- back, in the form of rationales for classification, during the labeling process (Zaidan et al., 2007, 2008; Donahue and Grauman, 2011; Sharma and Bilgic, 2018). For sentiment classification, for ex- ample, the annotators might highlight certain seg- ments of the text that convinced them to label the review as positive or negative (Figure 1). Unlike humans, a classifier …

Content truncated to 500 words