Bias at a Second Glance: A Deep Dive into Bias for German Educational Peer-Review Data Modeling

Proceedings of the 29th International Conference on Computational Linguistics, pages 1344–1356 October 12–17, 2022. 1344 Bias at a Second Glance: A Deep Dive into Bias for German Educational Peer-Review Data Modeling Thiemo Wambsganss∗1, Vinitra Swamy∗1, Roman Rietsche2, Tanja Käser1 1 EPFL, Lausanne, Switzerland {thiemo.wambsganss, vinitra.swamy, tanja.kaeser}@epfl.ch 2 University of St.Gallen, St.Gallen, Switzerland roman.rietsche@unisg.ch Abstract Natural Language Processing (NLP) has be- come increasingly utilized to provide adaptivity in educational applications. However, recent research has highlighted a variety of biases in pre-trained language models. While existing studies investigate bias in different domains, they are limited in addressing fine-grained anal- ysis on educational and multilingual corpora. In this work, we analyze bias across text and through multiple architectures on a corpus of 9,165 German peer-reviews collected from uni- versity students over five years. Notably, our corpus includes labels such as helpfulness, qual- ity, and critical aspect ratings from the peer- review recipient as well as demographic at- tributes. We conduct a Word Embedding As- sociation Test (WEAT) analysis on (1) our col- lected corpus in connection with the clustered labels, (2) the most common pre-trained Ger- man language models (T5, BERT, and GPT-2) and GloVe embeddings, and (3) the language models after fine-tuning on our collected data- set. In contrast to our initial expectations, we found that our collected corpus does not re- veal many biases in the co-occurrence analysis or in the GloVe embeddings. However, the pre-trained German language models find sub- stantial conceptual, racial, and gender bias and have significant changes in bias across concep- tual and racial axes during fine-tuning on the peer-review data. With our research, we aim to contribute to the fourth UN sustainability goal (quality education) with a novel dataset, an understanding of biases in natural language education data, and the potential harms of not counteracting biases in language models for educational tasks. 1 Introduction In recent years, Natural Language Processing (NLP) and Machine Learning (ML) have been ex- tensively used for improving adaptivity and individ- ualization of educational technology (Rosé et al., ∗ These authors contributed equally to this work. 2008; Xu et al., 2021). Researchers and practi- tioners have been developing a plethora of writing support systems (Song et al., 2014; Lauscher et al., 2018) and conversational agents (Ruan et al., 2019; Weber et al., 2021). More generally, there has been a rise in intelligent tutoring systems for educational purposes which provide learners adaptive feedback, e.g., on grammatical structures (White and Ro- zovskaya, 2020; Katinskaia and Yangarber, 2021; Kerz et al., 2021), language learning (Putra et al., 2021), argumentation (Song et al., 2014; Lauscher et al., 2019), or even empathy skills (Wambsganss et al., 2021). The technology for language-based personaliza- tion in education comes with a cost; a large body of research has been investigating and revealing biases in NLP systems (Bolukbasi et al., 2016; Sun et al., 2019). Bias has been found in multiple steps along the general NLP pipeline including the task setting, training data, pre-trained models (e.g. word embed- dings), and fine-tuned …

Content truncated to 500 words