Improvement of Statistical Machine Translation using Charater-Based Segmentationwith Monolingual and Bilingual Information

PACLIC 28 !145 Improvement of Statistical Machine Translation using Charater- Based Segmentation with Monolingual and Bilingual Information Vipas Sutantayawalee Peerachet Porkeaw Prachya Boonkwan Sitthaa Phaholphinyo Thepchai Supnithi National Electronics and Computer Technology Center, Thailand {vipas.sutantayawalee, peerachet.porkeaw,prachya.boonkwan, sitthaa.phaholphinyo,thepchai}@nectec.or.th Abstract We present a novel segmentation ap- proach for Phrase-Based Statistical Ma- chine Translation (PB-SMT) to languages where word boundaries are not obviously marked by using both monolingual and bi- lingual information and demonstrate that (1) unsegmented corpus is able to provide the nearly identical result compares to manually segmented corpus in PB-SMT task when a good heuristic character clus- tering algorithm is applied on it, (2) the performance of PB-SMT task has signifi- cantly increased when bilingual infor- mation are used on top of monolingual segmented result. Our technique, instead of focusing on word separation, mainly concentrate on a group of character. First, we group several characters that reside in an unsegmented corpus by employing pre- determined constraints and certain heuris- tics algorithms. Secondly, we enhance the segmented result by incorporating the character group repacking based on align- ment confidence. We evaluate the effec- tiveness of our method on PB-SMT task using English-Thai, English-Lao and Eng- lish-Burmese language pairs and report the best improvement of 8.1% increase in BLEU score on English-Thai pair. 1 Introduction Word segmentation is a crucial part of Statistical Machine Translation (SMT) especially for the lan- guages where there are no explicit word bounda- ries such as Chinese, Japanese, and Thai. The writing systems of these languages allow each word to be written consecutively without spaces between words. The issue of word boundary am- biguities arises if word boundary is misplaced, re- sulting in an incorrect translation. An effective word segmentator therefore becomes a crucial pre-processing step of SMT. Word segmentators which focusing on word which focusing on word, character [1] or both [2] and [3] have been imple- mented to accomplish this goal. Most of word segmentators are supervised; i.e. they require a monolingual corpus of a volu- minous size. Various approaches are employed, such as dictionary-based, Hidden Markov model (HMM), support vector machine (SVM), and con- ditional random field (CRF). Although, a number of segementators offer promising results, certain of them might be unsuitable for SMT task due to the influence of segmentation scheme [4]. There- fore, instead of solely rely on monolingual corpus, the use of a bilingual corpus as an guideline for word segmentation in improving the performance of SMT system has become of increasing interest [4] [5]. In this paper, we propose a novel segmenta- tion approach for Phrase-Based Statistical Ma- chine Translation (PB-SMT) to languages where word boundaries are not obviously marked by us- ing both monolingual and bilingual information on English-Thai, English-Burmese and English- Lao language pairs and demonstrate that (1) un- segmented corpus is able to provide the nearly identical result to manually segmented corpus in PB-SMT task when the good heuristics character clustering algorithm is applied on it, (2) the per- formance of PB-SMT task has significantly in- creased …

Content truncated to 500 words