Iterative Development and Evaluation of a Social Conversational Agent

International Joint Conference on Natural Language Processing, pages 1223–1229, Nagoya, Japan, 14-18 October 2013. Iterative Development and Evaluation of a Social Conversational Agent Annika Silvervarg, Arne J¨onsson Department of Computer and Information Science Link¨oping University, Link¨oping, Sweden annika.silvervarg@liu.se, arne.jonsson@liu.se Abstract We show that an agent with fairly good social conversational abilities can be built based on a limited number of topics and dialogue strategies if it is tailored to its intended users through a high degree of user involvement during an iterative devel- opment process. The technology used is pattern matching of question-answer pairs, coupled with strategies to handle: follow- up questions, utterances not understood, abusive utterances, repetitive utterances, and initiation of new topics. Introduction Social aspects of conversations with agents, such as small talk and narrative storytelling, can have a positive effect on peoples general interest in in- teracting with it and help build rapport (Bickmore, 2003). It can also be utilised to develop a relation- ship and establishing trust or the expertise of the agent (Bickmore and Cassell, 1999). We are in- terested in exploring if and how these and other effects transfer to an educational setting where children and teenagers interact with pedagogical agents in virtual learning environments. We see several reasons to incorporate social conversation with such agents, for example, it allows for cog- nitive rest, it can increase overall engagement and receptivity and it can make students feel more at ease with a learning task or topic (Silvervarg et al., 2010). There has, however, been few attempts to understand the users’ behaviour in social con- versations with pedagogical agents (Veletsianos and Russell, 2013) and embodied conversational agents (Robinson et al., 2008). In this paper we report on how we iteratively have worked with addressing the questions of 1) what do users talk about during social conversa- tion with a pedagogical agent, 2) how do users talk during social conversation with a pedagogical agent, 3) how does the answers to 1) and 2) affect the dialogue functions needed to implement social conversation with a pedagogical agent. A social conversational pedagogical agent Our work extends a virtual learning environ- ment with an educational math game named ”The Squares Family” (Pareto et al., 2009). A crucial part of the environment is a pedagogical agent, or more specifically a teachable agent (Biswas et al., 2001). While the student is playing the game, the agent ”learns” the rules of the game in two ways, by observation or through on-task multiple choice questions answered by the user. A teachable agent is independent and can act on its own, yet is de- pendent on the student to learn rules and strate- gies. The intended users are 12-14-year-old stu- dents, and the teachable agent is designed as hav- ing the same age or slightly younger. The conversational module for off-task conver- sations has been developed as a rather independent module of the learning environment. Off-task con- versation is based on a character description of the teachable agent that is consistent with the overall …

Content truncated to 500 words