Evaluating a Trainable Sentence Planner for a Spoken Dialogue System

Evaluating a Trainable Sentence Planner for a Spoken Dialogue System Owen Rambow AT&T Labs – Research Florham Park, NJ, USA rambow@research.att.com Monica Rogati Carnegie Mellon University Pittsburgh, PA, USA mrogati+@cs.cmu.edu Marilyn A. Walker AT&T Labs – Research Florham Park, NJ, USA walker@research.att.com Abstract Techniques for automatically training modules of a natural language gener- ator have recently been proposed, but a fundamental concern is whether the quality of utterances produced with trainable components can compete with hand-crafted template-based or rule- based approaches. In this paper We ex- perimentally evaluate a trainable sen- tence planner for a spoken dialogue sys- tem by eliciting subjective human judg- ments. In order to perform an ex- haustive comparison, we also evaluate a hand-crafted template-based genera- tion component, two rule-based sen- tence planners, and two baseline sen- tence planners. We show that the train- able sentence planner performs better than the rule-based systems and the baselines, and as well as the hand- crafted system. 1 Introduction The past several years have seen a large increase in commercial dialog systems. These systems typically use system-initiative dialog strategies, with system utterances highly scripted for style and register and recorded by voice talent. How- ever several factors argue against the continued use of these simple techniques for producing the system side of the conversation. First, text-to- speech has improved to the point of being a vi- able alternative to pre-recorded prompts. Second, there is a perceived need for spoken dialog sys- tems to be more flexible and support user initia- tive, but this requires greater flexibility in utter- ance generation. Finally, systems to support com- plex planning are being developed, which will re- quire more sophisticated output. As we move away from systems with pre- recorded prompts, there are two possible ap- proaches to producing system utterances. The first is template-based generation, where ut- terances are produced from hand-crafted string templates. Most current research systems use template-based generation because it is concep- tually straightforward. However, while little or no linguistic training is needed to write templates, it is a tedious and time-consuming task: one or more templates must be written for each combi- nation of goals and discourse contexts, and lin- guistic issues such as subject-verb agreement and determiner-noun agreement must be repeatedly encoded for each template. Furthermore, main- tenance of the collection of templates becomes a software engineering problem as the complexity of the dialog system increases.1 The second approach is natural language gen- eration (NLG), which customarily divides the generation process into three modules (Rambow and Korelsky, 1992): (1) Text Planning, (2) Sen- tence Planning, and (3) Surface Realization. In this paper, we discuss only sentence planning; the role of the sentence planner is to choose abstract lexico-structural resources for a text plan, where a text plan encodes the communicative goals for an utterance (and, sometimes, their rhetorical structure). In general, NLG promises portability across application domains and dialog situations by focusing on the development of rules for each generation module that are general and domain- 1Although …

Content truncated to 500 words