Evaluation of the Scusi? Spoken Language Interpretation System – A Case Study

International Joint Conference on Natural Language Processing, pages 225–233, Nagoya, Japan, 14-18 October 2013. Evaluation of the Scusi? Spoken Language Interpretation System – A Case Study Thomas Kleinbauer, Ingrid Zukerman and Su Nam Kim Faculty of Information Technology, Monash University Clayton, Victoria 3800, Australia Abstract We present a performance evaluation framework for Spoken Language Under- standing (SLU) modules, focusing on three elements: (1) characterization of spoken utterances, (2) experimental de- sign, and (3) quantitative evaluation met- rics. We then describe the application of our framework to Scusi?— our SLU sys- tem that focuses on referring expressions. 1 Introduction We present a performance evaluation framework for Spoken Language Understanding (SLU) mod- ules, and describe its application to the evaluation of Scusi? — an SLU system that focuses on the interpretation of descriptions of household objects (Zukerman et al., 2008). Our contributions pertain to (1) the characterization of spoken utterances, (2) experimental design, and (3) quantitative eval- uation metrics for an N-best list. Characterization of spoken utterances. Ac- cording to (Jokinen and McTear, 2010), “in diagnostic-type evaluations, a representative test suite is used so as to produce a system’s perfor- mance profile with respect to a taxonomy of pos- sible inputs”. In addition, one of the typical aims of an evaluation is to identify components that can be improved (Paek, 2001). These two factors in combination motivate a characterization of input utterances along two dimensions: accuracy and knowledge (Section 4). • Accuracy indicates whether an utterance de- scribes an intended object precisely and unam- biguously. For instance, when intending a blue plate, “the blue plate” is an accurate descrip- tion if there is only one such plate in the room, while “the green plate” is inaccurate. • Knowledge indicates how much the SLU mod- ule knows about different factors of the inter- pretation process, e.g., vocabulary or geometric relations. For instance, “CPU” in “the CPU un- der the desk”∗1 is Out of Vocabulary (OOV) for Scusi?, and the “of” in “the picture of a face”∗ is an unknown relation. The frequency of different values for these di- mensions influence the requirements from an SLU system, and the components that necessitate addi- tional resources, e.g., vocabulary extension. Experimental design. It is generally accepted that an SLU system should exhibit reasonable be- haviour by human standards. At present, in experi- ments that evaluate an SLU system’s performance, people speak to the system, and the accuracy of the system’s interpretation is assessed. However, this mode of evaluation, which we call Generative, does not address whether a system’s interpreta- tions are plausible (even if they are wrong). Thus, in addition to a Generative experiment, we offer an Interpretive experiment. Both experiments are briefly described below. Their implementation in our SLU system is described in Section 5. • In the Interpretive experiment, trial subjects and the SLU system are addressees, and are given utterances generated by a third party. The SLU system’s confidence in its interpretations is then compared with the preferences of the participants. • In …

Content truncated to 500 words