A Unified Tagging Approach to Text Normalization

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 688–695, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics A Unified Tagging Approach to Text Normalization Conghui Zhu Harbin Institute of Technology Harbin, China chzhu@mtlab.hit.edu.cn Jie Tang Department of Computer Science Tsinghua University, China jietang@tsinghua.edu.cn Hang Li Microsoft Research Asia Beijing, China hangli@microsoft.com Hwee Tou Ng Department of Computer Science National University of Singapore, Singapore nght@comp.nus.edu.sg Tiejun Zhao Harbin Institute of Technology Harbin, China tjzhao@mtlab.hit.edu.cn Abstract This paper addresses the issue of text nor- malization, an important yet often over- looked problem in natural language proc- essing. By text normalization, we mean converting ‘informally inputted’ text into the canonical form, by eliminating ‘noises’ in the text and detecting paragraph and sen- tence boundaries in the text. Previously, text normalization issues were often under- taken in an ad-hoc fashion or studied sepa- rately. This paper first gives a formaliza- tion of the entire problem. It then proposes a unified tagging approach to perform the task using Conditional Random Fields (CRF). The paper shows that with the in- troduction of a small set of tags, most of the text normalization tasks can be per- formed within the approach. The accuracy of the proposed method is high, because the subtasks of normalization are interde- pendent and should be performed together. Experimental results on email data cleaning show that the proposed method signifi- cantly outperforms the approach of using cascaded models and that of employing in- dependent models. 1 Introduction More and more ‘informally inputted’ text data be- comes available to natural language processing, such as raw text data in emails, newsgroups, fo- rums, and blogs. Consequently, how to effectively process the data and make it suitable for natural language processing becomes a challenging issue. This is because informally inputted text data is usually very noisy and is not properly segmented. For example, it may contain extra line breaks, extra spaces, and extra punctuation marks; and it may contain words badly cased. Moreover, the bounda- ries between paragraphs and the boundaries be- tween sentences are not clear. We have examined 5,000 randomly collected emails and found that 98.4% of the emails contain noises (based on the definition in Section 5.1). In order to perform high quality natural lan- guage processing, it is necessary to perform ‘nor- malization’ on informally inputted data first, spe- cifically, to remove extra line breaks, segment the text into paragraphs, add missing spaces and miss- ing punctuation marks, eliminate extra spaces and extra punctuation marks, delete unnecessary tokens, correct misused punctuation marks, restore badly cased words, correct misspelled words, and iden- tify sentence boundaries. Traditionally, text normalization is viewed as an engineering issue and is conducted in a more or less ad-hoc manner. For example, it is done by us- ing rules or machine learning models at different levels. In natural language processing, several is- sues of text normalization were studied, but were only done separately. This paper aims to conduct a thorough investiga- tion …

Content truncated to 500 words