A Question of Style: A Dataset for Analyzing Formality on Different Levels

Findings of the Association for Computational Linguistics: EACL 2023, pages 580–593 May 2-6, 2023 ©2023 Association for Computational Linguistics A Question of Style: A Dataset for Analyzing Formality on Different Levels Elisabeth Eder and Ulrike Krieg-Holz and Michael Wiegand Universität Klagenfurt, Klagenfurt, Austria {elisabeth.eder | ulrike.krieg-holz | michael.wiegand}@aau.at Abstract Accounting for different degrees of formality is crucial for producing contextually appropri- ate language. To assist NLP applications con- cerned with this problem and formality anal- ysis in general, we present the first dataset of sentences from a wide range of genres as- sessed on a continuous informal-formal scale via comparative judgments. It is the first corpus with a comprehensive perspective on German sentence-level formality overall. We compare machine learning models for formality scoring, a task we treat as a regression problem, on our dataset. Finally, we investigate the relation be- tween sentence- and document-level formality and evaluate leveraging sentence-based annota- tions for assessing formality on documents. 1 Introduction Textual style can be approached from various points of view. We focus on its inherent formality dimension stretching from informal to formal lan- guage use. See these two sentences, for example: (1) We gave thorough thought to an adequate example. Wir haben gründlich über ein adäquates Beispiel nachgedacht. (2) racked our brains about a niiice example... :D haben uns den kopf über ein schööönes beispiel zermartert... :D While both sentences transport the same content, they differ in their degree of formality. (2) is less formal than (1). It may be suitable only for more informal discourse contexts and inappropriate in formal settings. Understanding these different nu- ances of formality is crucial for effective commu- nication. Consequently, striking the right tone is relevant not only for humans but also for various NLP applications. May it be machine translation in need to transfer expressions of formality between different languages adequately (Niu and Carpuat, 2020; Anastasopoulos et al., 2022), chatbots aim- ing to produce contextually appropriate language to increase user satisfaction (Chaves et al., 2019; Elsholz et al., 2019), or writing assistance systems altering content to be more formal (Saberi et al., 2020). Hence, intra-lingual formality style transfer, which deals with generating a formal phrase given its informal version and vice versa, has recently also received increased attention (e.g., Shang et al., 2019 or Zhang et al., 2020). Our paper addresses a prerequisite for this task: assessing linguistic formality. Rating the trans- ferred style strength is necessary for evaluating formality style transfer models. Further, parallel corpora with formal and informal language pairs, often the basis for style transfer, are commonly built by automatically grading and extracting in- formal sentences first (Rao and Tetreault, 2018; Briakou et al., 2021b). For facilitating such formal- ity assessments and analyzing linguistic formality in general, we make the following contributions: 1. We present the first dataset of sentences from a wide range of genres with human formal- ity assessments on a continuous informal-formal scale. We ensure a comprehensive perspective on formality by collecting sentences from diverse do- mains. Formality annotations …

Content truncated to 500 words