Opportunities and Challenges in Neural Dialog Tutoring

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2357–2372 May 2-6, 2023 ©2023 Association for Computational Linguistics Opportunities and Challenges in Neural Dialog Tutoring Jakub Macina∗1,2 Nico Daheim∗3 Lingzhi Wang4 Tanmay Sinha5 Manu Kapur5 Iryna Gurevych3 Mrinmaya Sachan1 1Department of Computer Science, ETH Zürich 2 ETH AI Center 3 Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), TU Darmstadt 4 The Chinese University of Hong Kong 5Professorship for Learning Sciences and Higher Education, ETH Zürich jakub.macina@ai.ethz.ch Abstract Designing dialog tutors has been challenging as it involves modeling the diverse and com- plex pedagogical strategies employed by hu- man tutors. Although there have been signifi- cant recent advances in neural conversational systems using large language models (LLMs) and growth in available dialog corpora, dia- log tutoring has largely remained unaffected by these advances. In this paper, we rigorously analyze various generative language models on two dialog tutoring datasets for language learn- ing using automatic and human evaluations to understand the new opportunities brought by these advances as well as the challenges we must overcome to build models that would be usable in real educational settings. We find that although current approaches can model tu- toring in constrained learning scenarios when the number of concepts to be taught and possi- ble teacher strategies are small, they perform poorly in less constrained scenarios. Our hu- man quality evaluation shows that both models and ground-truth annotations exhibit low per- formance in terms of equitable tutoring, which measures learning opportunities for students and how engaging the dialog is. To understand the behavior of our models in a real tutoring setting, we conduct a user study using expert annotators and find a significantly large num- ber of model reasoning errors in 45% of con- versations. Finally, we connect our findings to outline future work. https://github.com/eth-nlped/ dialog-tutoring 1 Introduction The goal of dialog tutoring research is to build sys- tems that can tutor students using natural language conversation (Wollny et al., 2021). For several decades, learning scientists have been studying the *Equal contribution. features of domain-specific dialog tutoring systems that engender learning in students (Chi et al., 1994; Graesser et al., 1995; Moore et al., 2004; Litman et al., 2006; Graesser, 2016; Ruan et al., 2019) and have established strong learning gains that are even comparable to human tutoring in specific do- mains (Nye et al., 2014). However, these systems require extensive authoring of materials by teach- ers (MacLellan and Koedinger, 2020) and therefore cannot fully utilize the scalability of online learn- ing. Building dialog tutors is technically challeng- ing as tutoring dialogs typically exhibit properties that are absent in other forms of dialog. Tutoring dialogs are often long, enabling students to be ex- posed to the concepts in a way that they can use them in future (Chi and Wylie, 2014), and grounded in the learning scenarios (Graesser et al., 2009). Fi- nally, good dialog tutors are engaging and create …

Content truncated to 500 words