Assessing the Retrieval Effectiveness of a Speech Retrieval System by Simulating Recognition Errors

Assessing the Retrieval Effectiveness of a Speech :Retrieval System by Simulating Recognition Errors Peter Sch~uble, Ulrike Glavitsch Swiss Federal Institute of Technology (ETH) CH-8092 Zurich, Switzerland ABSTRACT We show how the recognition performance of a speech recog- nition component in a speech retrieval system affects the retrieval effectiveness. A speech retrieval system facilitates content-based retrieval of speech documents, i.e. audio recordings containing spoken text. The speech retrieval pro- cess receives queries from users and for every query it ranks the speech documents in decreasing order of their probabil- ities that they are relevant to the query. The speech recog- nition component is an important part of a speech retrieval system, since it detects the occurrences of indexing features in the documents. Because the recognition of indexing fea- tures in continuous speech is error prone, the question arises how much an error prone recognition of indexing features af- fects the retrieval effectiveness. As an answer to this question and main contribution of this paper we simulated the recog- nition of indexing features in speech documents on standard information retrieval test collections and show the resulting retrieval accuracies. 1. Introduction We show how the recognition performance of a speech recog- nition component in a speech retrieval system affects the retrieval effectiveness. A speech retrieval system facilitates content-based retrieval of speech documents, i.e. audio recordings containing spoken text [5]. The speech retrieval process receives queries from users and for every query it ranks the speech documents in decreasing order of their prob- abilities that they are relevant to the query. These probabil- ities are derived from the occurrences of indexing features that were identified in the speech documents by a speech recognition component [4]. Because the recognition of index- ing features in continuous speech is error prone, the question arises how much an error prone recognition of indexing fea- tures affects the retrieval effectiveness. The indexing features used in our speech retrieval system are phonetically motivated subword units having an inter- mediate specificity. The general pattern of an indexing fea- ture is a maximum sequence of consonants enclosed by two maximum sequences of vowels at both ends. We call these indexing features VCV-features where C stands for the maxi- mum sequence of consonants and V stands for the maximum sequence of vowels. As an example, the word INTERNA- TIONAL contains the VCV-features INTE, ERNA, ATIO, and IONA. The indexing vocabulary is defined to be the set of those VCV-featutes ~,i whose inverse document frequen- cies idf(~oi) are between a lower bound idfmi, and an up- per bound idf, no, such that the indexing features are neither 370 very specific nor very broad. The lower bound guarantees the suitability for indexing and the upper bound guarantees the trainability, i.e. there are enough examples to train the 8MMs. Experiments on standard information retrieval test collections showed that when using an appropriate subset of only 1000 VCV-features we can achieve a retrieval effective- ness that is comparable to standard weighted retrieval which is based on a …

Content truncated to 500 words