HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12557–12572 July 9-14, 2023 ©2023 Association for Computational Linguistics HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation Qianyu He1, Yikai Zhang1, Jiaqing Liang2∗, Yuncheng Huang1, Yanghua Xiao1∗, Yunwen Chen3 1Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University 2School of Data Science, Fudan University 3DataGrand Inc., Shanghai, China {qyhe21, ykzhang22, yunchenghuang22}@m.fudan.edu.cn, {liangjiaqing, shawyh}@fudan.edu.cn, chenyunwen@datagrand.com Abstract Similes play an imperative role in creative writing such as story and dialogue generation. Proper evaluation metrics are like a beacon guiding the research of simile generation (SG). However, it remains under-explored as to what criteria should be considered, how to quantify each criterion into metrics, and whether the metrics are effective for comprehensive, effi- cient, and reliable SG evaluation. To address the issues, we establish HAUSER, a holistic and automatic evaluation system for the SG task, which consists of five criteria from three per- spectives and automatic metrics for each cri- terion. Through extensive experiments, we verify that our metrics are significantly more correlated with human ratings from each per- spective compared with prior automatic metrics. Resources of HAUSER are publicly available at https://github.com/Abbey4799/HAUSER. 1 Introduction Similes play a vital role in human expression, mak- ing literal sentences imaginative and graspable. For example, Robert Burns famously wrote “My Luve is like a red, red rose” to metaphorically depict the beloved as being beautiful. In this simile, “Luve” (a.k.a. topic) is compared with “red rose” (a.k.a. ve- hicle) via the implicit property “beautiful” and the event “is”. Here, topic, vehicle, property, and event are four main simile components (Hanks, 2013). As a figure of speech, similes have been widely used in literature and conversations (Zheng et al., 2019; Chakrabarty et al., 2022). Simile generation (SG) is a crucial task in natu- ral language processing (Chakrabarty et al., 2020; Zhang et al., 2021; Lai and Nissim, 2022), with the aim of polishing literal sentences into simi- les. In Fig. 1, the literal sentence “He yelps and howls.” is polished into a simile by inserting the phrase “like a wolf”, resulting in “He yelps and ∗Corresponding author. Literal Sentence: He yelps and howls . GroundTruth: He yelps and howls like a wolf. 1. He yelps and howls like a wounded buffalo. 2. He yelps and howls like a chicken. 3. He yelps and howls like hell. Generated Simile Candidates … Human BLEU HAUSER Quality Top1 : #1 Creativity Top1: #1 Informativeness Top1 : #1 Quality Top1 : #2 : : what else criteria should I use… #1 The most high-quality and is Figure 1: An example of Simile Generation (SG) Evaluation. The commonly used automatic metric BLEU deems the sec- ond candidate as the most high-quality one among all the generated similes, while our proposed metrics HAUSER deem the first candidate as the best one regarding its quality, creativ- ity and informativeness, which better correlates with human ratings and also provides more criteria for SG evaluation. …

Content truncated to 500 words