User studies and the design of Natural Language Systems

User studies and the design of Natural Language Systems Steve Whittaker and Phil Stenton Hewlett-Packard Laboratories Filton Road, Bristol BS12 6QZ, UK. email: sjw~hplb.hpl.hp.com Abstract This paper presents a critical discussion of the vari- ous approaches that have been used in the evaluation of Natural Language systems. We conclude that pre- vious approaches have neglected to evaluate systems in the context of their use, e.g. solving a task requir- ing data retrieval. This raises questions about the validity of such approaches. In the second half of the paper, we report a laboratory study using the Wizard of Oz technique to identify NL requirements for carry- ing out this task. We evaluate the demands that task dialogues collected using this technique, place upon a prototype Natural Language system. We identify three important requirements which arose from the task that we gave our subjects: operators specific to the task of database access, complex contextual refer- ence and reference to the structure of the information source. We discuss how these might be satisfied by future Natural Language systems. 1 Introduction 1.1 Approaches to the evaluation of NL systems It is clear that a number of different criteria might be employed in the evaluation of Natural Language (NL) systems. It is also clear that there is no consensus on how evaluation should be carried out [RQR*88, GM84]. Among the different criteria that have been suggested are (a) Coverage; (b) Learnabil- ity; (c) General software requirements; (d) Compar- ison with other interface media. Coverage is con- cerned with the set of inputs which the system should be capable of handling and one issue we will discuss is how this set should be identified. Learnability is premised on the fact that complete coverage is not forseeable in the near future. As a consequence, any NL system will have limitations and one problem for users will be to learn to communicate within such limitations. Learnability is measured by the ease with which new users are able to identify these cov- erage limitations, and exploit what coverage is avail- able to carry out their task. The general software criteria of importance are speed, size, modifiabil- ity and installation and maintenance costs. Com- parison studies have mainly required users to per- form the same task using either a formal query lan- guage such as SQL or a restricted natural language and evaluated one against the other on such param- eters as time to solution or number of queries per task[SW83, JTS*85]. Our discussion will mainly ad- dress the problem of coverage: we shall not discuss these other issues further. Our concern here will be with interactive NL in- terfaces and not other applications of NL technology such as MT or messaging systems. Interactive inter- faces are not designed to be used in isolation, rather, they are intended to be connected to some sort of backend system, to improve access to that system. Our view is that NL systems should be evaluated with this in mind: the aim will be …

Content truncated to 500 words