Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer

Proceedings of NAACL-HLT 2018, pages 129–140 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer Sudha Rao University of Maryland, College Park∗ raosudha@cs.umd.edu Joel Tetreault Grammarly joel.tetreault@grammarly.com Abstract Style transfer is the task of automatically trans- forming a piece of text in one particular style into another. A major barrier to progress in this field has been a lack of training and eval- uation datasets, as well as benchmarks and au- tomatic metrics. In this work, we create the largest corpus for a particular stylistic trans- fer (formality) and show that techniques from the machine translation community can serve as strong baselines for future work. We also discuss challenges of using automatic metrics. 1 Introduction One key aspect of effective communication is the accurate expression of the style or tone of some content. For example, writing a more persuasive email in a marketing position could lead to in- creased sales; writing a more formal email when applying for a job could lead to an offer; and writ- ing a more polite note to your future spouse’s par- ents, may put you in a good light. Hovy (1987) argues that by varying the style of a text, people convey more information than is present in the lit- eral meaning of the words. One particularly im- portant dimension of style is formality (Heylighen and Dewaele, 1999). Automatically changing the style of a given content to make it more formal can be a useful addition to any writing assistance tool. In the field of style transfer, to date, the only available dataset has been for the transformation of modern English to Shakespeare, and it led to the application of phrase-based machine transla- tion (PBMT) (Xu et al., 2012) and neural machine translation (NMT) (Jhamtani et al., 2017) models to the task. The lack of an equivalent or larger dataset for any other form of style transfer has blocked progress in this field. Moreover, prior ∗This research was performed when the first author was at Grammarly. work has mainly borrowed metrics from machine translation (MT) and paraphrase communities for evaluating style transfer. However, it is not clear if those metrics are the best ones to use for this task. In this work, we address these issues through the following three contributions: • Corpus: We present Grammarly’s Yahoo Answers Formality Corpus (GYAFC), the largest dataset for any style containing a to- tal of 110K informal / formal sentence pairs. Table 1 shows sample sentence pairs. • Benchmarks: We introduce a set of learning models for the task of formality style trans- fer. Inspired by work in low resource MT, we adapt existing PBMT and NMT approaches for our task and show that they can serve as strong benchmarks for future work. • Metrics: In addition to MT and paraphrase metrics, we evaluate our models along three axes: formality, fluency and meaning preser- vation …

Content truncated to 500 words