Structured and Extended Named Entity Evaluation in Automatic Speech Transcriptions

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 518–526, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Structured and Extended Named Entity Evaluation in Automatic Speech Transcriptions Olivier Galibert1 Sophie Rosset2 Cyril Grouin2 Pierre Zweigenbaum2 Ludovic Quintard1 1LNE, Trappes, France first.last@lne.fr2 LIMSI–CNRS, Orsay, France first.last@limsi.fr Abstract The evaluation of named entity recogni- tion (NER) methods is an active field of research. This includes the recognition of named entities in speech transcripts. Evaluating NER systems on automatic speech recognition (ASR) output whereas human reference annotation was prepared on clean manual transcripts raises difficult alignment issues. These issues are empha- sized when named entities are structured, as is the case in the Quaero NER challenge organized in 2010. This paper describes the structured named entity definition used in this challenge and presents a method to transfer reference annotations to ASR out- put. This method was used in the Quaero 2010 evaluation of extended named entity annotation on speech transcripts, whose results are given in the paper. 1 Introduction Named Entity Detection has been studied since the MUC conferences in 1987. The notion has been extended to deal with mono- or multi-word ex- pressions that belong to a potentially interesting class for an application. Given a set of entity defi- nitions and a natural language corpus, systems try to extract and categorize all the relevant occurring entities. These entities can be used to feed further systems such as Information Retrieval, Question- Answering, Distillation, Terminology studies, etc. Traditional Named Entity Recognition (NER) is a task where proper nouns and numerical ex- pressions are extracted from documents and clas- sified into categories (person, location, organiza- tion, date, etc.). As shown by Voorhees and Har- man (2000), it is a key technology of Information Extraction (IE) and Open-Domain Question An- swering. NER is also used as a fundamental com- ponent in a variety of language processing appli- cations such as text clustering, topic detection, and summarization. While significant progress has been reported on the NER task, most of these approaches have gen- erally focused on clean textual data such as Sang and Meulder (2003). In the mean time, Kubala et al. (1998), Palmer et al. (1999), Turmo et al. (2009) and many others have focused on speech data. Named Entity detection evaluation over French spoken data has been proposed within the Ester II project, as described by Galliano et al. (2009). Within the framework of the Quaero project,we proposed an extended named entity definition with compositional and hierarchical structure. This ex- tension raises new issues and challenges in NER evaluation. First, as we shall explain below in more detail, the usual evaluation methods can- not compute the Slot Error Rate (SER) metric when named entities are compositional and recur- sive. Second, following Burger et al. (1998) and Hirschman et al. (1999), we consider that the eval- uation of named entity recognition on noisy text output by automatic speech recognition (ASR) systems should take as reference the named enti- ties found …

Content truncated to 500 words