Analyzing Gender Bias in Student Evaluations

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 868–876, Osaka, Japan, December 11-17 2016. Analyzing Gender Bias in Student Evaluations Andamlak Terkik1, Emily Prud’hommeaux2, Cecilia O. Alm2, Christopher M. Homan1, Scott Franklin3 1Golisano College of Computing & Information Sciences, Rochester Institute of Technology 2College of Liberal Arts, Rochester Institute of Technology 3College of Science, Rochester Institute of Technology {†at3616|†emilypx|†coagla|§cmh|†svfsps}@{†rit.edu|§cs.rit.edu} Abstract University students in the United States are routinely asked to provide feedback on the quality of the instruction they have received. Such feedback is widely used by university administrators to evaluate teaching ability, despite growing evidence that students assign lower numerical scores to women and people of color, regardless of the actual quality of instruction. In this paper, we analyze students’ written comments on faculty evaluation forms spanning eight years and five STEM disciplines in order to determine whether open-ended comments reflect these same biases. First, we apply sentiment analysis techniques to the corpus of comments to determine the overall affect of each comment. We then use this information, in combination with other features, to explore whether there is bias in how students describe their instructors. We show that while the gender of the evaluated instructor does not seem to affect students’ expressed level of overall satisfaction with their instruction, it does strongly influence the language that they use to describe their instructors and their experience in class. 1 Introduction Student evaluations of teachers (SETs), in which students are asked to provide their assessment of the quality of instruction they have received in a particular course, have been in use for over a century. At the end of a course, students are given forms, in paper or electronic format, containing a series of questions about the course and instructor, some requiring Likert-type scale responses and others seeking free text responses. SETs have increasingly become the de facto standard for evaluating university-level teaching performance (Centra and Gaubatz, 2000). The impact of these surveys on faculty is enormous, as they affect tenure, promotion, and compensation decisions. Despite playing an outsized role in assessing teaching effectiveness, SETs have numerous shortcom- ings as tools for this task. Students, for example, are understandably often not well equipped to determine an instructor’s knowledge of the subject matter, and they can be unreliable judges of how well they have mastered material, one important and generally accepted measure of teaching effectiveness. Perhaps more troublingly, several studies have demonstrated biases, whether conscious or unconscious, in stu- dents’ evaluations of women instructors and instructors of color. Most previous research in this area has focused on numerical (more precisely, ordinal categorical) ratings of various qualities related to teaching effectiveness. In this paper, we instead focus on compu- tational analysis of the text responses to open-ended questions found in SETs. In the first part of this paper, we present a supervised instructor satisfaction classifier trained to identify the satisfaction polarity of SET comments. Next, we apply this model to a much larger dataset to examine how …

Content truncated to 500 words