tRuEcasIng

tRuEcasIng Lucian Vlad Lita ♠ Carnegie Mellon llita@cs.cmu.edu Abe Ittycheriah IBM T.J. Watson abei@us.ibm.com Salim Roukos IBM T.J. Watson roukos@us.ibm.com Nanda Kambhatla IBM T.J. Watson nanda@us.ibm.com Abstract Truecasing is the process of restoring case information to badly-cased or non- cased text. This paper explores truecas- ing issues and proposes a statistical, lan- guage modeling based truecaser which achieves an accuracy of ∼98% on news articles. Task based evaluation shows a 26% F-measure improvement in named entity recognition when using truecasing. In the context of automatic content ex- traction, mention detection on automatic speech recognition text is also improved by a factor of 8. Truecasing also en- hances machine translation output legibil- ity and yields a BLEU score improvement of 80.2%. This paper argues for the use of truecasing as a valuable component in text processing applications. 1 Introduction While it is true that large, high quality text corpora are becoming a reality, it is also true that the digital world is flooded with enormous collections of low quality natural language text. Transcripts from var- ious audio sources, automatic speech recognition, optical character recognition, online messaging and gaming, email, and the web are just a few exam- ples of raw text sources with content often produced in a hurry, containing misspellings, insertions, dele- tions, grammatical errors, neologisms, jargon terms ♠ Work done at IBM TJ Watson Research Center etc. We want to enhance the quality of such sources in order to produce better rule-based systems and sharper statistical models. This paper focuses on truecasing, which is the process of restoring case information to raw text. Besides text rEaDaBILiTY, truecasing enhances the quality of case-carrying data, brings into the pic- ture new corpora originally considered too noisy for various NLP tasks, and performs case normalization across styles, sources, and genres. Consider the following mildly ambiguous sen- tence “us rep. james pond showed up riding an it and going to a now meeting”. The case-carrying al- ternative “US Rep. James Pond showed up riding an IT and going to a NOW meeting” is arguably better fit to be subjected to further processing. Broadcast news transcripts contain casing errors which reduce the performance of tasks such as named entity tagging. Automatic speech recognition produces non-cased text. Headlines, teasers, section headers - which carry high information content - are not properly cased for tasks such as question answer- ing. Truecasing is an essential step in transforming these types of data into cleaner sources to be used by NLP applications. “the president” and “the President” are two viable surface forms that correctly convey the same infor- mation in the same context. Such discrepancies are usually due to differences in news source, authors, and stylistic choices. Truecasing can be used as a normalization tool across corpora in order to pro- duce consistent, context sensitive, case information; it consistently reduces expressions to their statistical canonical form. In this paper, we attempt to show the benefits of truecasing in general as a valuable building block for NLP applications rather …

Content truncated to 500 words