Integrating Natural Language Components into Graphical Discourse

Integrating Natural Language Components into Graphical Discourse Stephan Dilley, John Bateman, Ulrich Thiel, Anne Tissen GMD, Integrated Publication and Information Systems Institute (IPSI) DolivostralSe 15, D-6100 Darmstadt, FRG e-maih { bateman,dilley,thiel,tissen }@darmstadt.gmd.de Abstract In our current research into the design of cognitively well-mo- tivated interfaces relying primarily on the display of graphical information, we have observed that graphical information alone does not provide sufficient support to users particu- larly when situations arise that do not simply conform to the users' expectations. This can occur due to too much informa- tion being requested, too little, information of the wrong kind, etc. To solve this problem, we are working towards the integra- tion of natural language generation to augment the interaction functionalities of the interface. This is intended to support the generation of flexible natural language utterances which pin- point possible problems with a user's request and which fur- ther go on to outline the user's most sensible courses of action away from the problem. In this paper, we describe our first pro- totype, where we combine the graphical and interaction plan- ning capabilities of our graphical information system SIC! with the text generation capabilities of the Penman system. We illustrate the need for such a combined system, and also give examples of how a general natural language facility beneficial- ly augments the user's ability to navigate a knowledge base graphically. I Introduction Natural language interfaces to information systems allow the user to converse with the system in a natural way without being restricted by the syntax of a formal query language. However, users often have difficulties in stating their information need precisely (cf. Brooks et al., 1986) and so a user interface design emphasizing multimedia aspects of information systems may be more appropriate. An object oriented presentation meets this requirement because users are allowed to investigate in- formation items directly. Thus, the distance between the users' intentions and the objects they involve is reduced to a mini- mum (cf. Hutchins et al., 1986). However, visual presentations of abslract information items have to be designed carefully to support the user in perceiving them correctly. Often the in- herent limitations of available presentation forms, e.g. the number ofdisplayable items, are an impediment to an adequate visualization. These difficulties which are well known in visu- al languages can be attacked by combining the presentation of graphical objects to the user with cooperative system re- sponses in natural languages. Our approach to human-computer interaction provides the basis for an integration of different interaction styles, in our case natural language and graphics, in a multi-modal informa- tion system. The choice of an appropriate combination of modes is essential for a successful interface design. As there is not yet a complete theory of multi-modal interaction, this ques- tion has to be answered experimentally. Therefore, we propose a modular architecture for multi-modal interfaces composed of interaction tools. In this paper, we discuss the addition ot natural language generation to the interaction tools provided. This capability is embodied in …

Content truncated to 500 words