StyLEx: Explaining Style Using Human Lexical Annotations

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2843–2856 May 2-6, 2023 ©2023 Association for Computational Linguistics StyLEx: Explaining Style Using Human Lexical Annotations Shirley Anugrah Hayati Kyumin Park Dheeraj Rajagopal∗ Lyle Ungar Dongyeop Kang University of Minnesota KAIST Carnegie Mellon University University of Pennsylvania hayat023@umn.edu pkm9403@kaist.ac.kr dheeraj@cs.cmu.edu ungar@cis.upenn.edu dongyeop@umn.edu Abstract Large pre-trained language models have achieved impressive results on various style classification tasks, but they often learn spuri- ous domain-specific words to make predictions (Hayati et al., 2021). While human explanation highlights stylistic tokens as important features for this task, we observe that model explana- tions often do not align with them. To tackle this issue, we introduce StyLEx, a model that learns from human annotated explanations of stylistic features and jointly learns to perform the task and predict these features as model ex- planations. Our experiments show that StyLEx can provide human-like stylistic lexical expla- nations without sacrificing the performance of sentence-level style prediction on both in- domain and out-of-domain datasets. Explana- tions from StyLEx show significant improve- ments in explanation metrics (sufficiency, plau- sibility) and when evaluated with human an- notations. They are also more understandable by human judges compared to the widely-used saliency-based explanation baseline.1 1 Introduction People use style as a strategic choice for their per- sonal or social goals in communications, making style analysis a long-studied field in NLP (Hovy, 1987; Kabbara and Cheung, 2016; Kang and Hovy, 2021). While large language models have obtained state-of-the-art results on many NLP tasks, they have been shown to overfit to spurious correlations in data across several datasets (Sen et al., 2021; Schlangen, 2021; Bras et al., 2020). Hayati et al. (2021) found a phenomenon in style classification tasks where the model’s word-level explanation do not align with human’s stylistic cues (stylistic cues are words that signify the style of a text). For in- stance, words such as “performances” and “wrench” ∗ currently at Google 1Code and data are publicly available at https://github. com/minnesotanlp/stylex : StyLEx : Integrated Gradient : Both all the performances are top notch and once you get through the accents all or nothing becomes an emotional though still positive wrench of a sit StyLEx … with top grade … 0 1 1 positive Sentiment: .. top notch .. Explanation: 󰘋 󰪐 󰘆 Human annotation StyLEx context-aware stylistic lexica Human-curated stylistic lexicon dictionary emotional …. top fantastic awesome positive notch lovely Figure 1: StyLEx classifies the input sentence’s style and provides lexical explanation. Compared to expla- nations computed by the integrated gradient method (Mudrakarta et al., 2018), StyLEx can find more accu- rate stylistic words. Green highlight refers to human’s annotated positive word, pink for StyLEx, blue for base- line, and purple for both StyLEx and the baseline. in Figure 1 are marked as important cues for senti- ment by a saliency method. However, they are dif- ferent from words that humans perceive as essential features for predicting the style (“top,” “notch”). Prior research in style …

Content truncated to 500 words