Statistical Analysis on Large Scale Chinese Short Message Corpus and Automatic Short Message Error Correction

Statistical Analysis on Large Scale Chinese Short Message Corpus and Automatic Short Message Error Correction Rile Hu 1, Yuezhong Tang 1, Chen Li 1,2 and Xia Wang 1 1NOKIA Research Center, Beijing 2Beijing University of Posts and Telecommunications {rile.hu, yuezhong.tang, ext-chen.9.li, xia.s.wang}@nokia.com Abstract. Analysis of short message corpus is an important foundation for research of automatic short message processing technology. Based on large scale short message corpus, this paper firstly presents statistical data and performs analysis in detail on basic information of short message corpus and special language phenomena in it. The distributions of the corpus parameters and special language phenomena are also given out. The statistical results presented in the paper are meaningful for research of robust short message understanding and implementation of short message based man- machine dialog system and short message based machine translation system. And we also build an automatic error correction system on mobile phone to correct the misapplication of Chinese character in short messages. The preliminary results show that our method is effective. Keywords: computer application; Chinese information processing; corpus technology; statistical analysis; short message; error correction. 1. Introduction With the development and popularization of mobile phones, the short message is widely used for communication and plays more and more important role in people’s daily life. As a main part of mobile communication, the short message has become a hot spot of mobile communication service in China at present because of its huge quantity and rapid progress. According to the statistics, there are 0.4 billion mobiles and 300 billion short messages in 2005 in China. The short message is so widely used and rapidly communicated that it makes our lives much convenient. However, on the other hand, there are a series of problems on the use and application of the short message. The limitation of mobile input leads to wrong words, self-made symbols, non-punctuation and many other non-standard Chinese problems in short message text at present. These non-standard words and application customs spread to all ages and many places with the fast and wide radiation of short messages, which makes a huge negative effect to correctly use and inherit Chinese. (For example, the students form all over the country use many short message words to write compositions of the entrance examination to college.) With the development of Corpus Linguistics (Kennedy, 2000), more and more corpora are built for research works. The results of the research works are widely used in natural language processing tasks. For example, Brown University in US built BROWN corpus, Lancaster University in UK and Oslo University and Bergen University in Norway built LOB corpus (Johansson, 1991). In China, large scale corpora were built by Tsinghua University, Peking University, Chinese Academy of Sciences and Chinese Academy of Social Sciences. Statistic and analysis had been done on these corpora (Feng,1999) (Feng ,2002) (Yu, 1998) (Yu,2002) (Guo, 2005) (Hu, 2002). The works listed above are focused on written languages. There are also some research works on Chinese spoken language. Qualitative analysis had been …

Content truncated to 500 words