Improving Text Normalization using Character-Blocks Based Models and System Combination

Proceedings of COLING 2012: Technical Papers, pages 1587–1602, COLING 2012, Mumbai, December 2012. Improving Text Normalization Using Character-blocks based Models and System Combination ChenLi Yang Liu Department of Computer Science The University of Texas at Dallas 800 W. Campbell Road, Richardson, Texas {chenli,yangl}@hlt.utdallas.edu ABSTRACT There are many abbreviation and non-standard tokens in SMS and Twitter messages. Nor- malizing these non-standard tokens will ease natural language processing modules for these domains. Recently, character-level machine translation (MT) and sequence labeling methods have been used for this normalization task, and demonstrated competitive performance. In this paper, we propose an approach to segment words into blocks of characters according to their phonetic symbols, and apply MT and sequence labeling models on such block-level. We also propose to combine these methods, as well as with other existing methods, in order to leverage their different strengths. Our experiments show our proposed approach achieved high precision and broad coverage. TITLE AND ABSTRACT IN CHINESE 使 使 使用 用 用字 字 字符 符 符块 块 块模 模 模型 型 型和 和 和系 系 系统 统 统合 合 合成 成 成提 提 提高 高 高文 文 文字 字 字纠 纠 纠错 错 错率 率 率 在手机短信和Twitter消息中存在许多缩写和非规范的单词。纠正这些非规范的单词 会给后续的自然语言处理模型带来便利。近年来，基于字符层面的机器翻译方法和序列标 注方法被广泛应用到这个任务中，并且大大提高了纠错率。本文介绍了一种根据单词发音 来切割单词的方法，然后把上述的两种模型利用在切割后的字符块层面进行纠错尝试。另 外，我们还尝试了将现有的方法和提出的方法进行各种合成以期利用各自的优点来提高最 终的效果。最后我们的实验表明我们的方法在对非规范词的纠错召回率和准确率上都有了 明显的提高。 KEYWORDS: text normalization, machine translation, CRF, NLP application. KEYWORDS IN CHINESE: 文本纠错，机器翻译，条件随机场，自然语言处理应用. 1587 1 Introduction There has been a rapid increase in social text in the last few years, including the mobile phone text messages (SMS), comments from the social media websites such as Facebook and Twitter, and real-time communication platforms like MSN and Gtalk. For example, by July 20121, Twitter has more than 500 million users, and more than 340 million new tweets are sent everyday. This trend attracts a lot of research in order to mine valuable information and knowledge from this data. Unfortunately, traditional NLP tools sometimes perform poorly when processing this kind of text. One of reasons is that social text is very informal, and contains many misspelled words, abbreviations and many other non-standard tokens. There are different efforts to develop robust language processing techniques for this domain. One is to develop NLP tools that are designed specially for tweets or SMS. For example, the system in (Ritter et al., 2011) reduced the errors of POS tagging by 41% compared with Stanford POS Tagger, and by 22% in parsing tweets compared with OpenNLP chunker tool. Another way is to perform some preprocessing on the original informal text, such as normalization, such that the resulting text is easier for the subsequent modules. The task of normalization is to automatically convert the non-standard tokens into corresponding standard words. Intuitively this will ease subsequent language processing modules in this domain. For example, if ‘2mr’ is converted to ‘tomorrow’, a text-to-speech system will know how to pronounce it, a POS tagger can label it correctly, and an information extraction system can identify it as a time expression. Non-standard tokens are formed in different ways in social text, and because of different rea- sons, including factors such as …

Content truncated to 500 words