Less is More: Attention Supervision with Counterfactuals for Text Classification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6695–6704, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6695 Less is More: Attention Supervision with Counterfactuals for Text Classification Seungtaek Choi Yonsei University hist0613@yonsei.ac.kr Haeju Park Yonsei University phj0225@yonsei.ac.kr Jinyoung Yeo Yonsei University jinyeo@yonsei.ac.kr Seung-won Hwang Yonsei University seungwonh@yonsei.ac.kr Abstract We aim to leverage human and machine in- telligence together for attention supervision. Specifically, we show that human annotation cost can be kept reasonably low, while its quality can be enhanced by machine self- supervision. Specifically, for this goal, we ex- plore the advantage of counterfactual reason- ing, over associative reasoning typically used in attention supervision. Our empirical results show that this machine-augmented human at- tention supervision is more effective than exist- ing methods requiring a higher annotation cost, in text classification tasks, including sentiment analysis and news categorization. 1 Introduction The practical importance of attention mechanism has been well-established, for both (a) improving NLP models (Vaswani et al., 2017), and also (b) en- hancing human understanding of models (Serrano and Smith, 2019; Wiegreffe and Pinter, 2019). This paper pursues the former direction, but un- like existing models, typically using attention in “unsupervised” nature. Adding human supervision to attention has been shown to improve model pre- dictions and explanations (Jain and Wallace, 2019). For example, consider a review in (Tang et al., 2019) “this place is small and crowded but the service is quick”. Models with unsupervised at- tention may attend highly on “quick”, a generic strong signal for restaurant reviews, but one may supervise to focus on “crowded” to guide models to predict a negative sentiment correctly. For this goal, attention supervision task (Yu et al., 2017; Liu et al., 2017) treats attention as output variables so that models can be trained to generate similar attention to human supervision. We cate- gorize such human supervision into the following two levels: • Sample level rationale: In the above exam- ple, whether to attend on quick or crowded depends on the ground-truth sentiment class. Human annotator is required to examine each training sample, and highlight important words specific to a sample and its class label. • Task level: An alternative with lower anno- tation overhead would be annotating vocabu- lary, separately from training samples. That is, both quick and crowded are annotated to attend, since both have high importance for the target task of sentiment classification. A naive belief would be assuming the former with a higher annotation cost is more effective at supervising the model’s attention. Our key claim, in contrast, is that requiring more annotation, or, sample-specific supervision, can be less effective than requiring less from human then augmenting it by machine (less-is-more-hypothesis). Similar skepticism on asking more, or sample-level ratio- nales from humans, was explored in (Bao et al., 2018), where machine attention from large addi- tional annotations was more effective supervisions than rationales. In this paper, we validate less-is-more without additional annotation overhead, by proposing a holistic approach of combining both human an- …

Content truncated to 500 words