Exploring Distantly-Labeled Rationales in Neural Network Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5571–5582 August 1–6, 2021. ©2021 Association for Computational Linguistics 5571 Exploring Distantly-Labeled Rationales in Neural Network Models Quzhe Huang, Shengqi Zhu, Yansong Feng∗, Dongyan Zhao Wangxuan Institute of Computer Technology, Peking University, China The MOE Key Laboratory of Computational Linguistics, Peking University, China {huangquzhe, zhusq, fengyansong, zhaody}@pku.edu.cn Abstract Recent studies strive to incorporate various human rationales into neural networks to im- prove model performance, but few pay atten- tion to the quality of the rationales. Most existing methods distribute their models’ fo- cus to distantly-labeled rationale words en- tirely and equally, while ignoring the potential important non-rationale words and not distin- guishing the importance of different rationale words. In this paper, we propose two novel auxiliary loss functions to make better use of distantly-labeled rationales, which encourage models to maintain their focus on important words beyond labeled rationales (PINs) and al- leviate redundant training on non-helpful ra- tionales (NoIRs). Experiments on two repre- sentative classification tasks show that our pro- posed methods can push a classification model to effectively learn crucial clues from non- perfect rationales while maintaining the ability to spread its focus to other unlabeled important words, thus significantly outperform existing methods. 1 Introduction Recent studies have shown an increasing interest in incorporating human knowledge into neural net- work models (Xu et al., 2018; Vashishth et al., 2018; Luo et al., 2018; Li and Srikumar, 2019; Jiang et al., 2020). For many natural language processing (NLP) tasks, such domain knowledge often refers to salient words annotated by human experts, which are also called rationales. Table 1 (top) shows an example of expert-annotated ratio- nales for sentiment analysis, which highlight note- worthy tokens and score the contributions of these tokens. The detailed annotations reflect the impor- tance of these words from the expert annotator’s viewpoint and are expected to help training better sentiment classification models. ∗Corresponding author. Expert Labeled Painful[0.1] to watch, but[0.7] viewers willing to take a chance will be rewarded[0.6] with two of the year’s most accomplished[0.6] and riveting[0.9] film performance. Distantly Labeled ::::: Painful to watch, but viewers willing to take a chance will be rewarded with two of the year’s most accomplished and ::::: riveting film per- formance. Table 1: An example of rationale annotation for senti- ment analysis. Words in underline are rationales anno- tated by human experts, and ::::: words in wavy underline are annotated via sentiment lexicon matching. Num- bers in [] are salience scores labeled by experts. Nonetheless, careful, case-by-case rationale an- notations inevitably involve large amounts of man- ual efforts, and are often extravagant or not even available. In practice, distantly-labeled rationales serve as a plausible alternative. Instead of labelling case by case, annotators could design heuristic rules to generate rationales for the whole dataset. For instance, in sentiment analysis, annotators can collect words with strong sentiment polarity (posi- tive or negative) to construct a sentiment lexicon, …

Content truncated to 500 words