Actions Speak Louder than Words: Evaluating Parsers in the Context of Natural Language Understanding Systems for Human-Robot Interaction

Proceedings of Recent Advances in Natural Language Processing, pages 56–62, Hissar, Bulgaria, 12-14 September 2011. Actions Speak Louder than Words: Evaluating Parsers in the Context of Natural Language Understanding Systems for Human-Robot Interaction Sandra K¨ubler, Rachael Cantrell, Matthias Scheutz Indiana University {skuebler,rcantrel,mscheutz}@indiana.edu Abstract The standard ParsEval metrics alone are often not sufficient for evaluating parsers integrated in natural language understand- ing systems. We propose to augment in- trinsic parser evaluations by extrinsic mea- sures in the context of human-robot inter- action using a corpus from a human co- operative search task. We compare a con- stituent with a dependency parser on both intrinsic and extrinsic measures and show that the conversion to semantics is feasible for different syntactic paradigms. 1 Introduction Human-robot interactions (HRI) in natural lan- guage (Scheutz et al., 2007) pose many chal- lenges for natural language understanding (NLU) systems, for humans expect robots to (1) gener- ate quick responses to their request, which re- quires all processing to be done in real-time, (2) to rapidly integrate perceptions (e.g., to resolve refer- ents (Brick and Scheutz, 2007)), and (3) to provide backchannel feedback indicating whether they un- derstood an instruction, often before the end of an utterance. As a result, NLU systems on robots must operate incrementally to allow for the con- struction of meaning that can lead to robot action before an utterance is completed (e.g., a head-turn of the robot to check for an object referred to by the speaker). Hence, the question arises how one can best evaluate NLU components such as parsers for robotic NLU in the context of HRI. In this paper, we argue that intrinsic parser eval- uations, which evaluate parsers in isolation, are insufficient for determining their performance in HRI contexts where the ultimate goal of the NLU system is to generate the correct actions for the robot in a timely manner. For high performance of a parser with respect to intrisic measures does not imply that the parser will also work well with the other NLU components. A correct but overly com- plex parse passed to the semantic analysis unit, for example, may not result in the correct meaning interpretation and will thus fail to generate cor- rect actions. Similarly, fragmented input from the speech recognizer may not lead to any parsable se- quence of words, again likely resulting in incorrect robot behavior. Hence, we need an extrinsic eval- uation to determine the utility and performance of a parser in the context of other NLU components at the level of semantics and action execution. To this end, we introduce an evaluation archi- tecture that can be used for extrinsic evaluations of NLU components and demonstrate its utility for parser evaluation using state-of-the-art parsers for each of the two main parsing paradigms: the Berkeley constituent parser (Petrov and Klein, 2007) and MaltParser (Nivre et al., 2007b), a de- pendency parser. The evaluation compares in- trinsic and extrinsic measures on the CReST cor- pus (Eberhard et al., 2010), which is representa- tive …

Content truncated to 500 words