QuickSet: Multimodal Interaction for Simulation Set-up and Control

QuickSet: Multimodal Interaction for Simulation Set-up and Control Philip R. Cohen, Michael Johnston, David McGee, Sharon Oviatt, Jay Pittman, Ira Smith, Liang Chen and Josh Clow Center for Human Computer Communication Oregon Graduate Institute of Science and Technology P.O.Box 91000 Portland, OR 97291-1000 USA Tel: 1-503-690-1326 E-mail: pcohen@cse.ogi.edu http://www.cse.ogi.edu/CHCC ABSTRACT This paper presents a novel multimodal system applied to the setup and control of distributed interactive simulations. We have developed the QuickSet prototype, a pen/voice system running on a hand-held PC, communicating through a distributed agent architecture to NRaD's ~ LeatherNet system, a distributed interactive training simulator built for the US Marine Corps (USMC). The paper briefly describes the system and illustrates its use in multimodal simulation setup. KEYWORDS: multimodal interfaces, agent architecture, gesture recognition, speech recognition, natural language processing, distributed interactive simulation. 1. INTRODUCTION In order to train personnel more effectively, the US military is developing large-scale distributed simulation capabilities. Begun as SIMNET in the 1980's [23], these distributed, interactive environments attempt to provide a high degree of fidelity in simulating combat, including simulations of the individual combatants, the equipment, entity movements, atmospheric effects, etc. There are four general phases of user interaction with these simulations: Creating entities, supplying their initial behavior, interacting with the entities during a running simulation, and reviewing the results. The present research concentrates on the first two of these stages. Our contribution to the distributed interactive simulation (DIS) effort is to rethink the nature of the user interaction. As with most modern simulators, DISs are controlled via graphical user interfaces (GUIs). However, the simulation GUI is showing signs of strain, since even for a small-scale scenario, it requires users to choose from hundreds of entities in order to select the desired ones to place on a map. To compound these interface problems, the military is intending to increase the scale of the simulations I NRaD = US Navy Command and Control Ocean Systems Center Research Development Test and Evaluation (San Diego). 20 dramatically, while at the same time, for reasons of mobility and affordability, desiring that simulations should be creatable from small devices (e.g., PDAs). This impending collision of trends for smaller screen size and for more entities requires a different paradigm for human- computer interaction. We have argued generically that GUI technologies offer advantages in allowing users to manipulate objects that are on the screen, in reminding users of their options, and in minimizing errors [7]. However, GUIs are often weak in supporting interactions with many objects, or objects not on the screen. In contrast, it was argued that linguistically- based interface technologies offer the potential to describe large sets of objects, which may not all be present on a screen, and can be used to create more complex behaviors through specification of rule invocation conditions. Simulation is one type of application for which these limitations of GUIs, as well as the strengths of natural language, especially spoken language, are apparent [6]. It has become clear, however, that speech-only interaction is not optimal …

Content truncated to 500 words