Representation of Lexical Stylistic Features in Language Models’ Embedding Space

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 370–387 July 13-14, 2023 ©2023 Association for Computational Linguistics Representation of Lexical Stylistic Features in Language Models’ Embedding Space Qing Lyu Marianna Apidianaki Chris Callison-Burch University of Pennsylvania {lyuqing, marapi, ccb}@seas.upenn.edu Abstract The representation space of pretrained Lan- guage Models (LMs) encodes rich information about words and their relationships (e.g., sim- ilarity, hypernymy, polysemy) as well as ab- stract semantic notions (e.g., intensity). In this paper, we demonstrate that lexical stylistic no- tions such as complexity, formality, and figu- rativeness, can also be identified in this space. We show that it is possible to derive a vector representation for each of these stylistic notions from only a small number of seed pairs. Using these vectors, we can characterize new texts in terms of these dimensions by performing simple calculations in the corresponding em- bedding space. We conduct experiments on five datasets and find that static embeddings en- code these features more accurately at the level of words and phrases, whereas contextualized LMs perform better on sentences. The lower performance of contextualized representations at the word level is partially attributable to the anisotropy of their vector space, which can be corrected to some extent using techniques like standardization.1 1 Introduction The style of a text is often reflected in its gram- matical and discourse properties, but also in local word choices made by the author. The choice of one from a set of synonyms or paraphrases with different connotations can define the style of a text in terms of complexity (e.g., help vs. assist), for- mality (e.g., dad vs. father), figurativeness (e.g., fall vs. plummet), and so on (Edmonds and Hirst, 2002). These lexical stylistic features can be useful in various scenarios, such as analyzing the style of authors or texts of different genres, and determining the appropriate word usage in language learning applications. 1Our code and data are publicly available at https:// github.com/veronica320/Lexical-Stylistic -Features. COMPLEXITY doctor → medical practitioner laws → legislative texts high blood pressure → hypertension very common → prevalent a lot → significant quantity be bad → impact negatively help → assist FORMALITY my gosh → jesus breathing → respiratory yeah → yes ten years → decade first of all → foremost a whole bunch → full my dad → father FIGURATIVENESS bright → radiant heavy → burdened unsympathetic → cold-hearted fall → plummet a lot of → a sea of quick → lightning hard → ironclad Table 1: Seed pairs for constructing vector represen- tations of complexity (simple → complex), formality (casual → formal), and figurativeness (literal → figura- tive). Previous approaches to formality detection relied on word length, frequency, as well as on the pres- ence of specific prefixes and suffixes (e.g., intra-, -ation) (Brooke et al., 2010). Such features have also been used for complexity detection, often com- bined with information regarding the number of word senses and synonyms (Shardlow, 2013; Kriz et al., 2018). Recent studies have shown …

Content truncated to 500 words