Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1638–1653 August 1–6, 2021. ©2021 Association for Computational Linguistics 1638 Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions Dorottya Demszky1 Jing Liu2 Zid Mancenido3 Julie Cohen4 Heather Hill3 Dan Jurafsky1 Tatsunori Hashimoto1 1Stanford University 2University of Maryland 3Harvard University 4University of Virginia {ddemszky, thashim}@stanford.edu Abstract In conversation, uptake happens when a speaker builds on the contribution of their in- terlocutor by, for example, acknowledging, re- peating or reformulating what they have said. In education, teachers’ uptake of student con- tributions has been linked to higher student achievement. Yet measuring and improving teachers’ uptake at scale is challenging, as ex- isting methods require expensive annotation by experts. We propose a framework for com- putationally measuring uptake, by (1) releas- ing a dataset of student-teacher exchanges ex- tracted from US math classroom transcripts annotated for uptake by experts; (2) formal- izing uptake as pointwise Jensen-Shannon Di- vergence (PJSD), estimated via next utterance classification; (3) conducting a linguistically- motivated comparison of different unsuper- vised measures and (4) correlating these mea- sures with educational outcomes. We find that although repetition captures a significant part of uptake, PJSD outperforms repetition- based baselines, as it is capable of identifying a wider range of uptake phenomena like ques- tion answering and reformulation. We apply our uptake measure to three different educa- tional datasets with outcome indicators. Un- like baseline measures, PJSD correlates signifi- cantly with instruction quality in all three, pro- viding evidence for its generalizability and for its potential to serve as an automated profes- sional development tool for teachers.1 1 Introduction Building on the interlocutor’s contribution via, for example, acknowledgment, repetition or elabora- tion (Figure 1), is known as uptake and is key to a successful conversation. Uptake makes an inter- locutor feel heard and fosters a collaborative inter- action (Collins, 1982; Clark and Schaefer, 1989), 1Code and annotated data: https://github.com/ ddemszky/conversational-uptake I added 30 to 70… Okay. Good, you did the first step. Okay, you added 30 to 70. And you got what? acknowledgment collaborative completion t1 t2 t3 t4 repetition reformulation s Where did the 70 come from? t5 elaboration Figure 1: Example student utterance s and possible teacher replies t, illustrating different uptake strategies. which is especially important in contexts like edu- cation. Teachers’ uptake of student ideas promotes dialogic instruction by amplifying student voices and giving them agency in the learning process, un- like monologic instruction where teachers lecture at students (Bakhtin, 1981; Wells, 1999; Nystrand et al., 1997). Despite extensive research showing the positive impact of uptake on student learning and achievement (Brophy, 1984; O’Connor and Michaels, 1993; Nystrand et al., 2003), measuring and improving teachers’ uptake at scale is challeng- ing as existing methods require manual annotation by experts and are prohibitively resource-intensive. We introduce a framework for computationally measuring uptake. First, we create and release a dataset of 2246 student-teacher exchanges ex- tracted from …

Content truncated to 500 words