Romanization-based Approach to Morphological Analysis in Korean SMS Text Processing

International Joint Conference on Natural Language Processing, pages 145–152, Nagoya, Japan, 14-18 October 2013. Romanization-based Approach to Morphological Analysis in Korean SMS Text Processing Youngsam Kim Seoul National University/ Gwanak-1, Gwanak-ro, Gwanak-gu, Seoul, South Korea youngsamy@gmail.com Hyopil Shin Seoul National University/ Gwanak-1, Gwanak-ro, Gwanak-gu, Seoul, South Korea hpshin@snu.ac.kr Abstract In this research, we suggest an approach to retrieval-related tasks for Korean SMS text. Most of the previous approaches to such text used morphological analysis as the routine stage of the preprocessing workflow, functionally equivalent to POS tagging. However, such approaches suffer difficulties since Short Message Service language usually contains irregular or- thography, atypically spelled words, un- spaced segments, etc. Two experiments were conducted to measure how well these problems can be avoided with the transliteration of Korean to Roman letters. In summary, we will argue that such a Romanization-based retrieval method has several advantages since it provides an easier way to preprocess the data with a variety of linguistic rules. 1 Introduction In this internet era, everyday people express opinions, comments, or sentiments; all of which can be accessed via the web. Particularly with the popularization of mobile computing devices, it has become easier than ever for people to share messages using social media services like Twit- ter or Facebook. However, such an environment brings new challenges for researchers who aim to analyze or interpret this linguistic data. One of the problems they encounter is that these written texts have a different form than those in pub- lished books or articles. They were often called as short message service language, txt-speak, chat-speak, etc. This new data source has re- ceived attentions from various fields and re- searchers working in the field of sentiment anal- ysis and opinion-mining often find that dealing with such texts using traditional approaches is problematic. For agglutinative languages like Korean, since words are formed by combining lemmas and var- ious affixes, morphological analysis is required to find the functional meaning of each compo- nent. Most previous studies used morphological analysis only to preprocess the text, but this ap- proach exhibits several weaknesses when used on the data that is written in SMS-like languages. First of all, texts are often unspaced to save on typing time and sentence length (e.g., Twitter only allows 140 characters per tweet). Secondly, many words are not typed in the same way as their dictionary entries; the letters are changed or reduced to smaller units due to morpho-phonetic variation and abbreviation processes. This paper will propose a new approach to overcome these shortcomings for morphological- ly rich languages while making use of Korean case studies. This approach adopts Yale Romani- zation to transliterate Korean alphabets into Ro- man letters, which, due to the way it handles Ko- rean characters, allows for a more intuitive and easier way of implementing the relevant rewrit- ing rules and handling morph-phonetic changes. In Section 2, the problems of morphological analysis will be described and the properties of Korean SMS language will be reviewed. This will be …

Content truncated to 500 words