Revisiting non-English Text Simplification: A Unified Multilingual Benchmark

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 428–446 July 9-14, 2023 ©2023 Association for Computational Linguistics Knowledge of cultural moral norms in large language models Aida Ramezani Department of Computer Science University of Toronto armzn@cs.toronto.edu Yang Xu Department of Computer Science Cognitive Science Program University of Toronto yangxu@cs.toronto.edu Abstract Moral norms vary across cultures. A recent line of work suggests that English large lan- guage models contain human-like moral bi- ases, but these studies typically do not exam- ine moral variation in a diverse cultural setting. We investigate the extent to which monolin- gual English language models contain knowl- edge about moral norms in different countries. We consider two levels of analysis: 1) whether language models capture fine-grained moral variation across countries over a variety of top- ics such as “homosexuality” and “divorce”; 2) whether language models capture cultural di- versity and shared tendencies in which top- ics people around the globe tend to diverge or agree on in their moral judgment. We per- form our analyses with two public datasets from the World Values Survey (across 55 coun- tries) and PEW global surveys (across 40 coun- tries) on morality. We find that pre-trained En- glish language models predict empirical moral norms across countries worse than the En- glish moral norms reported previously. How- ever, fine-tuning language models on the sur- vey data improves inference across countries at the expense of a less accurate estimate of the English moral norms. We discuss the rel- evance and challenges of incorporating cul- tural knowledge into the automated inference of moral norms. 1 Introduction Moral norms vary from culture to culture (Haidt et al., 1993; Bicchieri, 2005; Atari et al., 2022; Iurino and Saucier, 2020). Understanding the cul- tural variation in moral norms has become criti- cally relevant to the development of machine in- telligence. For instance, recent work has shown that cultures vary substantially in their judgment to- ward moral dilemmas regarding autonomous driv- ing (Awad et al., 2018, 2020). Work in Natural Language Processing (NLP) also shows that lan- guage models capture some knowledge of social or moral norms and values. For example, with no supervision, English pre-trained language mod- els (EPLMs) have been shown to capture people’s moral biases and distinguish between morally right and wrong actions (Schramowski et al., 2022). Here we investigate whether EPLMs encode knowl- edge about moral norms across cultures, an open issue that has not been examined comprehensively. Multilingual pre-trained language models (mPLMs) have been probed for their ability to identify cultural norms and biases in a restricted setting (Yin et al., 2022; Arora et al., 2022; Hämmerl et al., 2022; Touileb et al., 2022). For instance, Hämmerl et al. (2022) show that mPLMs capture moral norms in a handful of cultures that speak different languages. However, it remains unclear whether monolingual EPLMs encode cultural knowledge about moral norms. Prior studies have only used EPLMs to assess how they encode undesirable biases toward different communities …

Content truncated to 500 words