A Neural Few-Shot Text Classification Reality Check

42487

Authors

nan

Abstract

AbstractModern classification models tend to struggle when the amount of annotated data is scarce. To overcome this issue, several neural few-shot classification models have emerged, yielding significant progress over time, both in Computer Vision and Natural Language Processing. In the latter, such models used to rely on fixed word embeddings, before the advent of transformers. Additionally, some models used in Computer Vision are yet to be tested in NLP applications. In this paper, we compare all these models, first adapting those made in the field of image processing to NLP, and second providing them access to transformers. We then test these models equipped with the same transformer-based encoder on the intent detection task, known for having a large amount of classes. Our results reveal that while methods perform almost equally on the ARSC dataset, this is not the case for the Intent Detection task, where most recent and supposedly best competitors perform worse than older and simpler ones (while all are are given access to transformers). We also show that a simple baseline is surprisingly strong. All the new developed models as well as the evaluation framework are made publicly available.

Keywords

nan

Full Content

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 78–90 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 78 RESPER : Computationally Modelling Resisting Strategies in Persuasive Conversations. Ritam Dutt∗,1, Sayan Sinha∗,2, Rishabh Joshi1, Surya Shekhar Chakraborty3, Meredith Riggs1, Xinru Yan1, Haogang Bao1, Carolyn Penstein Ros´e1 1Carnegie Mellon University, 2Indian Institute of Technology Kharagpur, 3Zendrive Inc {rdutt,rjoshi2,mriggs,xinruyan,haogangb,cprose}@cs.cmu.edu, sayan.sinha@iitkgp.ac.in, suryaschak@gmail.com Abstract Modelling persuasion strategies as predictors of task outcome has several real-world appli- cations and has received considerable attention from the computational linguistics community. However, previous research has failed to ac- count for the resisting strategies employed by an individual to foil such persuasion attempts. Grounded in prior literature in cognitive and social psychology, we propose a generalised framework for identifying resisting strategies in persuasive conversations. We instantiate our framework on two distinct datasets comprising persuasion and negotiation conversations. We also leverage a hierarchical sequence-labelling neural architecture to infer the aforementioned resisting strategies automatically. Our experi- ments reveal the asymmetry of power roles in non-collaborative goal-directed conversations and the benefits accrued from incorporating resisting strategies on the final conversation outcome. We also investigate the role of dif- ferent resisting strategies on the conversation outcome and glean insights that corroborate with past findings. We also make the code and the dataset of this work publicly avail- able at https://github.com/americast/ resper. 1 Introduction Persuasion is pervasive in everyday human interac- tions. People are often exposed to scenarios that challenge their existing beliefs and opinions, such as medical advice, election campaigns, and adver- tisements (Knobloch-Westerwick and Meng, 2009; Bartels, 2006; Speck and Elliott, 1997). Of late, huge strides have been taken by the Computational Linguistics community to advance research in per- suasion. Some seminal works include identifying persuasive strategies in text (Yang et al., 2019) and conversations (Wang et al., 2019), investigating the ∗ denotes equal contribution interplay of language and prior beliefs on success- ful persuasion attempts (Durmus and Cardie, 2018; Longpre et al., 2019), and generating persuasive dialogues (Munigala et al., 2018). However, a relatively unexplored domain by the community is the investigation of resisting strate- gies employed to foil persuasion attempts. As suc- cinctly observed by Miller (1965): “In our daily lives we are struck not by the ease of producing at- titude change but by the rarity of it.” Several works in cognitive and social psychology (Fransen et al., 2015a; Zuwerink Jacks and Cameron, 2003) have put forward different resisting strategies and the motivations for the same. However, so far, there has not been any attempt to operationalise these strategies from a computational standpoint. We attempt to bridge this gap in our work. We propose a generalised framework, grounded in cognitive psychology literature, for automati- cally identifying resisting strategies in persuasion oriented discussions. We instantiate our framework on two publicly available datasets comprising per- suasion and negotiation conversations to create an annotated corpus of resisting strategies. Furthermore, we design a hierarchical sequence modelling framework, that leverages the conver- sational context to …

Content truncated to 500 words

Labeled Text

AbstractModern classification models tend to struggle when the amount of annotated data is scarce. To overcome this issue, several neural few-shot classification models have emerged, yielding significant progress over time, both in Computer Vision and Natural Language Processing. In the latter, such models used to rely on fixed word embeddings, before the advent of transformers. Additionally, some models used in Computer Vision are yet to be tested in NLP applications. In this paper, we compare all these models, first adapting those made in the field of image processing to NLP, and second providing them access to transformers. We then test these models equipped with the same transformer-based encoder on the intent detection task, known for having a large amount of classes. Our results reveal that while methods perform almost equally on the ARSC dataset, this is not the case for the Intent Detection task, where most recent and supposedly best competitors perform worse than older and simpler ones (while all are are given access to transformers). We also show that a simple baseline is surprisingly strong. All the new developed models as well as the evaluation framework are made publicly available.

Publication Details

Paper ID:
42487
Year:
2021
Publication:
TIPSTER
Publication Type:
conf
DOI:
nan