Investigating Robustness of Dialog Models to Popular Figurative Language Constructs

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7476–7485 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7476 Investigating Robustness of Dialog Models to Popular Figurative Language Constructs Harsh Jhamtani ∗ 1 Varun Gangal ∗ 1 Eduard Hovy 1 Taylor Berg-Kirkpatrick 2 1 School of Computer Science, Carnegie Mellon University 2 Computer Science and Engineering. University of California San Diego {jharsh,vgangal,hovy}@cs.cmu.edu, tberg@ucsd.eng.edu Abstract Humans often employ figurative language use in communication, including during interac- tions with dialog systems. Thus, it is impor- tant for real-world dialog systems to be able to handle popular figurative language constructs like metaphor and simile. In this work, we ana- lyze the performance of existing dialog models in situations where the input dialog context ex- hibits use of figurative language. We observe large gaps in handling of figurative language when evaluating the models on two open do- main dialog datasets. When faced with dia- log contexts consisting of figurative language, some models show very large drops in perfor- mance compared to contexts without figurative language. We encourage future research in di- alog modeling to separately analyze and report results on figurative language in order to better test model capabilities relevant to real-world use. Finally, we propose lightweight solutions to help existing models become more robust to figurative language by simply using an exter- nal resource to translate figurative language to literal (non-figurative) forms while preserving the meaning to the best extent possible. 1 Introduction Human frequently employ figurative language such as metaphors (Carbonell, 1982) and idioms (Jack- endoff, 1995) for effective and/or stylistic com- munication. Thus, dialog models interacting with humans should be equipped to handle these forms of communication. However, understanding figu- rative language might be challenging for machines since figurative constructions often exhibit non- compositional semantics and may rely on shared cultural and common-sense knowledge (Carbonell and Minton, 1983). For example, a powerful GPT2 model fine-tuned on DailyDialog dataset is unable to handle the metaphor ‘built on the sand’ (Figure ∗ HJ and VG contributed equally for this paper. Order decided by coin flip. Figure 1: An example illustrating how model responses are affected by figurative constructs in dialog context. Here, the model conflates the metaphorical use of build on the sand with its literal meaning, leading to an inappropriate, atopical response. 1), and the response seems to rely on the unintended literal sense of ‘sand’. In this work, we investigate the performance of existing dialog models when faced with inputs con- taining figurative language use. (1) First, we iden- tify the subsets in existing datasets (such as Daily- Dialog (Li et al., 2017) and PersonaChat (Zhang et al., 2018)) which have figurative language use such as metaphors and similes. We observe that the performance of all the dialog models under consideration is lower on such subsets containing figurative language use compared to the dataset as a whole. (2) Second, we gather manually writ- ten literal/non-figurative equivalents of the dialog utterances in DailyDialog and PersonaChat which exhibit figurative …

Content truncated to 500 words