Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7109–7130 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7109 Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework Anna Wegmann and Dong Nguyen Department of Information and Computing Sciences Utrecht University Utrecht, the Netherlands a.m.wegmann@uu.nl, d.p.nguyen@uu.nl Abstract Style is an integral part of natural language. However, evaluation methods for style mea- sures are rare, often task-specific and usu- ally do not control for content. We pro- pose the modular, fine-grained and content- controlled similarity-based STyle EvaLuation framework (STEL) to test the performance of any model that can compare two sentences on style. We illustrate STEL with two general dimensions of style (formal/informal and sim- ple/complex) as well as two specific charac- teristics of style (contrac’tion and numb3r sub- stitution). We find that BERT-based methods outperform simple versions of commonly used style measures like 3-grams, punctuation fre- quency and LIWC-based approaches. We in- vite the addition of further tasks and task in- stances to STEL and hope to facilitate the im- provement of style-sensitive measures. 1 Introduction Natural language is not only about what is said (i.e., content), but also about how it is said (i.e., linguistic style). Linguistic style and social con- text are highly interrelated (Coupland, 2007; Bell, 2013). For example, people can accommodate their linguistic style to each other based on so- cial power differences (Danescu-Niculescu-Mizil et al., 2012). Furthermore, linguistic style can influence perception, e.g., the persuasiveness of news (El Baff et al., 2020) or the success of pitches on crowdsourcing platforms (Parhankangas and Renko, 2017). As a result, style is relevant for natu- ral language understanding, e.g., in author profiling (Rao et al., 2010), abuse detection (Markov et al., 2021) or understanding conversational interactions (Danescu-Niculescu-Mizil and Lee, 2011). Addi- tionally, style can be important to address in natural language generation (Ficler and Goldberg, 2017), including identity modeling in dialogue systems (Li et al., 2016) and style preservation in machine trans- lation (Niu et al., 2017; Rabinovich et al., 2017). Figure 1: STEL Task Instance. Anchor 1 (A1) and anchor 2 (A2) and the alternative sentences 1 (S1) and 2 (S2) are split along the same style dimension (here: formal/informal). The sentences and anchors are para- phrases of each other. The STEL task is to order S1 and S2 to match A1-A2. Here, the correct order is S2-S1. There are several general evaluation benchmarks for different linguistic phenomena (e.g., Wang et al. (2018, 2019)) but less emphasis has been put on linguistic style. Nevertheless, natural language pro- cessing literature shows a variety of approaches for the evaluation of style measuring methods: They have been tested on whether they group texts by the same authors together (Hay et al., 2020; Bevendorff et al., 2020), whether they can correctly classify the style for ground truth datasets (Niu and Carpuat, 2017; Kang and Hovy, 2021) and whether ‘simi- lar style words’ are similarly represented (Akama et al., 2018). However, these evaluation approaches are (i) …

Content truncated to 500 words