Aggregation Improves Learning: Experiments in Natural Language Generation for Intelligent Tutoring Systems

Proceedings of the 43rd Annual Meeting of the ACL, pages 50–57, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics Aggregation improves learning: experiments in natural language generation for intelligent tutoring systems Barbara Di Eugenio and Davide Fossati and Dan Yu University of Illinois Chicago, IL, 60607, USA {bdieugen,dfossa1,dyu6}@uic.edu Susan Haller University of Wisconsin - Parkside Kenosha, WI 53141, USA haller@cs.uic.edu Michael Glass Valparaiso University Valparaiso, IN, 46383, USA Michael.Glass@valpo.edu Abstract To improve the interaction between students and an intelligent tutoring system, we devel- oped two Natural Language generators, that we systematically evaluated in a three way com- parison that included the original system as well. We found that the generator which intu- itively produces the best language does engen- der the most learning. Specifically, it appears that functional aggregation is responsible for the improvement. 1 Introduction The work we present in this paper addresses three issues: evaluation of Natural Language Generation (NLG) systems, the place of aggregation in NLG, and NL interfaces for Intelligent Tutoring Systems. NLG systems have been evaluated in various ways, such as via task efficacy measures, i.e., mea- suring how well the users of the system perform on the task at hand (Young, 1999; Carenini and Moore, 2000; Reiter et al., 2003). We also employed task efficacy, as we evaluated the learning that occurs in students interacting with an Intelligent Tutoring System (ITS) enhanced with NLG capabilities. We focused on sentence planning, and specifically, on aggregation. We developed two different feedback generation engines, that we systematically evaluated in a three way comparison that included the orig- inal system as well. Our work is novel for NLG evaluation in that we focus on one specific com- ponent of the NLG process, aggregation. Aggrega- tion pertains to combining two or more of the mes- sages to be communicated into one sentence (Reiter and Dale, 2000). Whereas it is considered an es- sential task of an NLG system, its specific contri- butions to the effectiveness of the text that is even- tually produced have rarely been assessed (Harvey and Carberry, 1998). We found that syntactic aggre- gation does not improve learning, but that what we call functional aggregation does. Further, we ran a controlled data collection in order to provide a more solid empirical base for aggregation rules than what is normally found in the literature, e.g. (Dalianis, 1996; Shaw, 2002). As regards NL interfaces for ITSs, research on the next generation of ITSs (Evens et al., 1993; Litman et al., 2004; Graesser et al., 2005) explores NL as one of the keys to bridging the gap between cur- rent ITSs and human tutors. However, it is still not known whether the NL interaction between students and an ITS does in fact improve learning. We are among the first to show that this is the case. We will first discuss DIAG, the ITS shell we are using, and the two feedback generators that we de- veloped, DIAG-NLP1and DIAG-NLP2. Since the latter is based on a corpus study, we …

Content truncated to 500 words