Named Entity Extraction from Noisy Input: Speech and OCR

Named Entity Extraction from Noisy Input: Speech and OCR David Miller, Scan Boisen, Richard Schwartz, Rebecca Stone, Ralph Weischedel BBN Technologies 70 Fawcett Street Cambridge, MA 02138 dmiller@bbn.com, boisen@bbn.com, schwartz@bbn.com, rwstone@bbn.com, weischedel@bbn.com Abstract In this paper, we analyze the performance of name finding in the context of a variety of automatic speech recognition (ASR) systems and in the context of one optical character recognition (OCR) system. We explore the effects of word error rate from ASR and OCR, performance as a function of the amount of training data, and for speech, the effect of out-of-vocabulary errors and the loss of punctuation and mixed case I Introduction Information extraction systems have traditionally been evaluated on online text with relatively few errors in the input. For example, this description of the Nominator system (Wacholder et al. 1997) would apply to several other systems: "We chose The Wall Street Journal corpus because it follows standard stylistic conventions, especially capitalization, which is essential for Nominator to work." The real-world challenge, however, is pointed out in Palmer and Day (1997): "It is also unknown how the existing high-scoring systems would perform on less well-behaved texts, such as single-case texts, non-newswire texts, or text obtained via optical character recognition (OCR)." In this paper we explore how performance degrades on noisy input, in particular on broadcast news (speech) and on newspaper (printed matter). Error rates of automatic speech recognizers (ASR) on broadcast news are still very high, e.g., 14-28% word error. Though character error can be very low for laser printer output, word error rates of 20% are possible for OCR systems applied to newsprint or low-quality printed matter. In this paper, we evaluate a learning algorithm, a hidden Markov model (HM), for named entity extraction applied to human transcripts of news, to transcripts without case or punctuation (perfect speech output), to errorful ASR output and to OCR output. Extracting information from noisy sources poses the following challenges, which are addressed in the paper. • Since speech recognizers do not generate mixed case nor punctuation, how much do case and punctuation contribute to recognizing names in English? (Section 3.) Note that these challenges also arise in languages without case to signal proper nouns (e.g., Chinese, German, Japanese), in mono-case English or informal English (e.g., emails). • How much will performance degrade with increasing error in the input? (Section 4.) • How does closed vocabulary recognition affect information extraction performance? (Section 5) • For the learning algorithm employed, how much training and effort are required? (Section 6) • How much do lists of names contribute to performance? (Section 7) 316 2 Algorithms and Data 2.1 Task Definition and Data The named entity (NE) task used for this evaluation requires the system to identify all named locations, named persons, named organizations, dates, times, monetary amounts, and percentages. The task definition is given in Chinchor, et al, (1998). For speech recognition, roughly 175 hours of news broadcasts (roughly 1.2m words of audio) were available from the National Institute for Science …

Content truncated to 500 words