Vietnamese Text Accent Restoration with Statistical Machine Translation

Vietnamese Text Accent Restoration With Statistical Machine Translation Luan-Nghia Pham Department of Information Technology Haiphong University Haiphong,Vietnam nghialuan@gmail.com Viet-Hong Tran University of Economic And Technical Industries Hanoi,Vietnam thviet@uneti.edu.vn Vinh-Van Nguyen University of Engineering and Technology Vietnam National University Hanoi,Vietnam vinhnv@vnu.edu.vn Abstract Vietnamese accentless texts exist on paral- lel with official vietnamese documents and play an important role in instant message, mobile SMS and online searching. Under- standing correctly these texts is not sim- ple because of the lexical ambiguity caused by the diversity in adding diacritics to a given accentless sequence. There have been some methods for solving the vietnamese accentless texts problem known as accent prediction and they have obtained promis- ing results. Those methods are usually based on distance matching, n-gram, dictionary of words and phrases and heuristic techniques. In this paper, we propose a new method solving the accent prediction. Our method combine the strength of previous methods (combining n-gram method and phrase dic- tionary in general). This method considers the accent predicting as statistical machine translation (SMT) problem with source lan- guage as accentless texts and target language as accent texts, respectively. We also im- prove quality of accent predicting by apply- ing some techniques such as adding dictio- nary, changing order of language model and tuning. The achieved result and the ability to enhance proposed system are obviously promising. 1 Introduction Accent predicting problem refers to the situation where accents are removed (e.g. by some email preprocessing systems), cannot be entered (e.g. by standard English keyboards), or not explicitly rep- resented in the text (e.g. in Arabic). We resolve the languages using Roman characters in writ- ing together with additional accent and diacriti- cal marks. These languages include European lan- guages such as Spanish and French and Asian lan- guages such as Chinese Pinyin and Vietnamese. Vietnamese accentless texts coexist with official Vietnamese texts and it is relatively common texts on the internet. Official Vietnamese language is a complex language with many accent (including acute, grave, hook, tilder, and dot-below) and Lat- inh alphabets. These are two inseparable compo- nents in Vietnamese. However, many Vietnamese choose to use accentless Vietnamese because it is easier and quickly to type. For example, a official Vietnamese sentence: chúng tôi sẽ bay tới Hà Nội vào chủ nhật (’We will fly to Hanoi on Sunday’) will be written as an Vietnamese accentless sen- tence as chung toi se bay toi Ha Noi vao chu nhat. Decoding such a sentence could be quite hard for both human and machine because of lexical ambi- guity. For instance, the accentless term "toi" can easily lead to confusion between the original Viet- namese "tôi" (‘we’) and the plausible alternative "tới" (‘to’). Nowadays, the application of information tech- nology to exchanging information is more and more popular. We daily receive many of emails, SMS but the majority of them are without ac- cents which may cause troubles for interpreting the meaning. Therefore, automatic accent restoration of accentless Vietnamese texts have many of ap- plications …

Content truncated to 500 words