Learning More Effective Dialogue Strategies Using Limited Dialogue Move Features

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 185–192, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Learning More Effective Dialogue Strategies Using Limited Dialogue Move Features Matthew Frampton and Oliver Lemon HCRC, School of Informatics University of Edinburgh Edinburgh, EH8 9LW, UK M.J.E.Frampton@sms.ed.ac.uk, olemon@inf.ed.ac.uk Abstract We explore the use of restricted dialogue contexts in reinforcement learning (RL) of effective dialogue strategies for infor- mation seeking spoken dialogue systems (e.g. COMMUNICATOR (Walker et al., 2001)). The contexts we use are richer than previous research in this area, e.g. (Levin and Pieraccini, 1997; Scheffler and Young, 2001; Singh et al., 2002; Pietquin, 2004), which use only slot-based infor- mation, but are much less complex than the full dialogue “Information States” ex- plored in (Henderson et al., 2005), for which tractabe learning is an issue. We explore how incrementally adding richer features allows learning of more effective dialogue strategies. We use 2 user simu- lations learned from COMMUNICATOR data (Walker et al., 2001; Georgila et al., 2005b) to explore the effects of differ- ent features on learned dialogue strategies. Our results show that adding the dialogue moves of the last system and user turns increases the average reward of the auto- matically learned strategies by 65.9% over the original (hand-coded) COMMUNI- CATOR systems, and by 7.8% over a base- line RL policy that uses only slot-status features. We show that the learned strate- gies exhibit an emergent “focus switch- ing” strategy and effective use of the ‘give help’ action. 1 Introduction Reinforcement Learning (RL) applied to the prob- lem of dialogue management attempts to find op- timal mappings from dialogue contexts to sys- tem actions. The idea of using Markov Deci- sion Processes (MDPs) and reinforcement learn- ing to design dialogue strategies for dialogue sys- tems was first proposed by (Levin and Pierac- cini, 1997). There, and in subsequent work such as (Singh et al., 2002; Pietquin, 2004; Scheffler and Young, 2001), only very limited state infor- mation was used in strategy learning, based al- ways on the number and status of filled informa- tion slots in the application (e.g. departure-city is filled, destination-city is unfilled). This we refer to as low-level contextual information. Much prior work (Singh et al., 2002) concentrated only on specific strategy decisions (e.g. confirmation and initiative strategies), rather than the full problem of what system dialogue move to take next. The simple strategies learned for low-level def- initions of state cannot be sensitive to (sometimes critical) aspects of the dialogue context, such as the user’s last dialogue move (DM) (e.g. request- help) unless that move directly affects the status of an information slot (e.g. provide-info(destination- city)). We refer to additional contextual infor- mation such as the system and user’s last di- alogue moves as high-level contextual informa- tion. (Frampton and Lemon, 2005) learned full strategies with limited ‘high-level’ information (i.e. the dialogue move(s) of the last user utter- ance) and only used a stochastic user simulation whose probabilities …

Content truncated to 500 words