EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English

Findings of the Association for Computational Linguistics: ACL 2022, pages 2811 - 2823 May 22-27, 2022 c⃝2022 Association for Computational Linguistics EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English Weicheng Ma, Samiha Datta, Lili Wang, and Soroush Vosoughi Department of Computer Science, Dartmouth College {weicheng.ma.gr, samiha.datta.23, lili.wang.gr, soroush.vosoughi}@dartmouth.edu Abstract While cultural backgrounds have been shown to affect linguistic expressions, existing natural language processing (NLP) research on culture modeling is overly coarse-grained and does not examine cultural differences among speakers of the same language. To address this problem and augment NLP models with cultural back- ground features, we collect, annotate, manu- ally validate, and benchmark EnCBP, a finer- grained news-based cultural background predic- tion dataset in English. Through language mod- eling (LM) evaluations and manual analyses, we confirm that there are noticeable differences in linguistic expressions among five English- speaking countries and across four states in the US. Additionally, our evaluations on nine syn- tactic (CoNLL-2003), semantic (PAWS-Wiki, QNLI, STS-B, and RTE), and psycholinguis- tic tasks (SST-5, SST-2, Emotion, and Go- Emotions) show that, while introducing cul- tural background information does not bene- fit the Go-Emotions task due to text domain conflicts, it noticeably improves deep learning (DL) model performance on other tasks. Our findings strongly support the importance of cul- tural background modeling to a wide variety of NLP tasks and demonstrate the applicability of EnCBP in culture-related research. 1 Introduction Psychological research has revealed that people from different cultural background behave dif- ferently in the ways they think (Nisbett et al., 2001), talk (Kim, 2002), write (Krampetz, 2005; Almuhailib, 2019; Kitano, 1990), and express emo- tions (Hareli et al., 2015; Sun et al., 2021; Acheam- pong et al., 2020). NLP researchers have ap- plied cultural background information to model differences in linguistic expressions across culture groups especially for psycholinguistic tasks 1, e.g., 1In this paper, we refer to NLP tasks reflecting the psy- chological states of people, e.g., sentiments and emotions, as psycholinguistic tasks. distributional perspective identification (Tian et al., 2021) and sentiment analysis (Sun et al., 2021). In prior research, culture groups are usually defined by official language (Tian et al., 2021) (e.g., US, UK, and India are considered part of the same cul- ture group) or, even more coarse-grained, by ideol- ogy (Imran et al., 2020) (e.g., “Western" countries and “Eastern" countries). These settings typically overlook the nuanced cultural differences across or within countries, and they do not provide useful information for modeling different language use behavior in mono-lingual contexts. To study culture-specific linguistic expressions in the same language and to apply culture-related knowledge to other NLP tasks, we build EnCBP, a cultural background prediction dataset in English. Following (Tambassi, 2018), we assume that lan- guage use patterns are more consistent inside each country or each district in a large country, e.g., states in the US. As such, we first construct news corpora by sampling news articles covering five frequently discussed and controversial topics from major news outlets in five English-speaking coun- tries …

Content truncated to 500 words