VALUE: Understanding Dialect Disparity in NLU
39119Authors
nan
Abstract
AbstractEnglish Natural Language Understanding (NLU) systems have achieved great performances and even outperformed humans on benchmarks like GLUE and SuperGLUE. However, these benchmarks contain only textbook Standard American English (SAE). Other dialects have been largely overlooked in the NLP community. This leads to biased and inequitable NLU systems that serve only a sub-population of speakers. To understand disparities in current models and to facilitate more dialect-competent NLU systems, we introduce the VernAcular Language Understanding Evaluation (VALUE) benchmark, a challenging variant of GLUE that we created with a set of lexical and morphosyntactic transformation rules. In this initial release (V.1), we construct rules for 11 features of African American Vernacular English (AAVE), and we recruit fluent AAVE speakers to validate each feature transformation via linguistic acceptability judgments in a participatory design manner. Experiments show that these new dialectal features can lead to a drop in model performance.
Keywords
Full Content
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 308 - 319 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MISC: A MIxed Strategy-Aware Model Integrating COMET for Emotional Support Conversation Quan Tu1∗ † , Yanran Li2∗, Jianwei Cui2, Bin Wang2, Ji-Rong Wen1,3 and Rui Yan1,3‡ 1Gaoling School of Artificial Intelligence, Renmin University of China 2Xiaomi AI Lab 3Beijing Academy of Artificial Intelligence 1{quantu,jrwen,ruiyan}@ruc.edu.cn 2{liyanran,cuijianwei,wangbin11}@xiaomi.com Abstract Applying existing methods to emotional sup- port conversation—which provides valuable as- sistance to people who are in need—has two major limitations: (a) they generally employ a conversation-level emotion label, which is too coarse-grained to capture user’s instant men- tal state; (b) most of them focus on expressing empathy in the response(s) rather than gradu- ally reducing user’s distress. To address the problems, we propose a novel model MISC, which firstly infers the user’s fine-grained emo- tional status, and then responds skillfully us- ing a mixture of strategy. Experimental re- sults on the benchmark dataset demonstrate the effectiveness of our method and reveal the benefits of fine-grained emotion understand- ing as well as mixed-up strategy modeling. Our code and data could be found in https: //github.com/morecry/MISC. 1 Introduction Empathy is the ability to perceive what others feel, think in their places and respond properly. It has a broad application scenarios to endow machines with the ability of empathy, including automatic psycho-therapist, intelligent customer service, em- pathetic conversational agents, and etc (Fitzpatrick et al., 2017; Shin et al., 2019; Ma et al., 2020). In this work, we focus on a special kind of human-computer empathetic conversation, i.e., emotional support conversation (Liu et al., 2021). Distinguishedly, emotional support conversation happens between a seeker and supporter, where the supporter aims to gradually reduce seeker’s distress as the conversation goes. This makes ex- isting approaches unsuitable for our setting for at least two reasons. Firstly, existing work on emo- tional chatting learns to predict user emotion us- ing a conversation-level emotion label, which is ∗Equal Contribution. †This work was done during internship at Xiaomi AI Lab. ‡Corresponding author: Rui Yan (ruiyan@ruc.edu.cn). coarse-grained and static to the conversation con- text (Rashkin et al., 2019; Lin et al., 2019c; Li et al., 2020a). However, emotion is complex and user emotion intensity will change during the de- veloping of the conversation (Liu et al., 2021). It is thus a necessity to tell seeker’s fine-grained mental state at each utterance. Secondly, most of empa- thetic chatbots are trained to respond emotionally in accordance with the predicted coarse-grained emotion class, without consideration on how to ad- dress the seeker’s emotional problem (De Graaf et al., 2012; Majumder et al., 2020; Xie and Park, 2021). Hence, they are deficient to apply for emo- tional support conversation whose goal is to help others work through the challenges they face. Figure 1: An Emotional Support Conversation Example. To tackle these issues, we propose a novel ap- proach MISC, a.k.a. MIxed Srategy-aware model integrating COMET for emotional support conver- sation. …
Content truncated to 500 words
Labeled Text
AbstractEnglish Natural Language Understanding (NLU) systems have achieved great performances and even outperformed humans on benchmarks like GLUE and SuperGLUE. However, these benchmarks contain only textbook Standard American English (SAE). Other dialects have been largely overlooked in the NLP community. This leads to biased and inequitable NLU systems that serve only a sub-population of speakers. To understand disparities in current models and to facilitate more dialect-competent NLU systems, we introduce the VernAcular Language Understanding Evaluation (VALUE) benchmark, a challenging variant of GLUE that we created with a set of lexical and morphosyntactic transformation rules. In this initial release (V.1), we construct rules for 11 features of African American Vernacular English (AAVE), and we recruit fluent AAVE speakers to validate each feature transformation via linguistic acceptability judgments in a participatory design manner. Experiments show that these new dialectal features can lead to a drop in model performance.
Publication Details
- Paper ID:
- 39119
- Year:
- 2022
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan