Semantic Coherence Scoring Using an Ontology

Semantic Coherence Scoring Using an Ontology Iryna Gurevych Rainer Malaka Robert Porzel Hans-Peter Zorn European Media Lab GmbH Schloss-Wolfsbrunnenweg 31c D-69118 Heidelberg, Germany � gurevych,malaka,porzel,zorn@eml.org � Abstract In this paper we present ONTOSCORE, a sys- tem for scoring sets of concepts on the basis of an ontology. We apply our system to the task of scoring alternative speech recognition hypotheses (SRH) in terms of their semantic coherence. We conducted an annotation exper- iment and showed that human annotators can reliably differentiate between semantically co- herent and incoherent speech recognition hy- potheses. An evaluation of our system against the annotated data shows that, it successfully classifies 73.2% in a German corpus of 2.284 SRHs as either coherent or incoherent (given a baseline of 54.55%). 1 Introduction Following Allen et al. (2001), we can distinguish be- tween controlled and conversational dialogue systems. Since controlled and restricted interactions between the user and the system increase recognition and understand- ing accuracy, such systems are reliable enough to be de- ployed in various real world applications, e.g. public transportation or cinema information systems. The more conversational a dialogue system becomes, the less pre- dictable are the users’ utterances. Recognition and pro- cessing become increasingly difficult and unreliable. Today’s dialogue systems employ domain- and discourse-specific knowledge bases, so-called ontologies, to represent the individual discourse entities as concepts, and their relations to each other. In this paper we present an algorithm for measuring the semantic coherence of sets of concepts against such an ontology. In the fol- lowing, we will show how the semantic coherence mea- surement can be applied to estimate how well a given speech recognition hypothesis (SRH) fits with respect to the existing knowledge representation, thereby providing a mechanism that increases the robustness and reliability of dialogue systems. In Section 2 we discuss the problem of scoring and classifying SRHs in terms of their semantic coherence followed by a description of our annotation experiment. Section 3 contains a description of the kind of knowledge representations employed by ONTOSCORE. We present the algorithm in Section 4, and an evaluation of the cor- responding system for scoring SRHs is given in Section 5. A conclusion and additional applications are given in Section 6. 2 Semantic Coherence and Speech Recognition Hypotheses 2.1 The Problem While a simple one-best hypothesis interface between au- tomatic speech recognition (ASR) and natural language understanding (NLU) suffices for restricted dialogue sys- tems, more complex systems either operate on n-best lists as ASR) output or convert ASR word graphs (Oerder and Ney, 1993) into n-best lists, given the distribution of acoustic and language model scores (Schwartz and Chow, 1990; Tran et al., 1996). For example, in our data a user expressed the wish to see a specific city map again, as:1 (1) IIch w¨urde would die the Karte map gerne like wiedersehen to see again Looking at two SRHs from the ensuing n-best list we found that Example (1a) constituted a suitable represen- tation of the utterance, whereas Example (1b) constituted a less …

Content truncated to 500 words