Unsupervised Bilingual Segmentation using MDL for Machine Translation

31st Pacific Asia Conference on Language, Information and Computation (PACLIC 31), pages 89–96 Cebu City, Philippines, November 16-18, 2017 Copyright c⃝2017 Bin Shan, Hao Wang and Yves Lepage Unsupervised Bilingual Segmentation using MDL for Machine Translation Bin Shan, Hao Wang, Yves Lepage Graduate School of Information, Production and Systems Waseda University 2-7 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0135, Japan {reynolds@fuji., oko ips@ruri., yves.lepage@}waseda.jp Abstract In statistical machine translation systems, a problem arises from the weak performance in alignment due to differences in word form or granularity across different languages. To address this problem, in this paper, we pro- pose a unsupervised bilingual segmentation method using the minimum description length (MDL) principle. Our work aims at improv- ing translation quality using a proper segmen- tation model (lexicon). For generating bilin- gual lexica, we implement a heuristic and it- erative algorithm. Each entry in this bilingual lexicon is required to hold a proper length and the ability to fit the data well. The results show that this bilingual segmentation signifi- cantly improved the translation quality on the Chinese–Japanese and Japanese–Chinese sub- tasks. 1 Introduction Words are generally the smallest processing units in varieties of NLP tasks. However, there is no guarantee that such smallest processing units can fit any NLP tasks. Especially in bilingual tasks (e.g. statistical machine translation), different lan- guages have different writing systems or segmen- tation granularity. Such problem should be consid- ered as a critical factor of performance in transla- tion quality. For instance, in machine translation experiments on 11 Europarl corpora (Koehn, 2005), Finnish has the lowest translation accuracy as eval- uated by BLEU scores when translated into En- glish. French–Spanish has the highest BLEU scores. Finnish is a non-Indo-European and agglutinative language. French and Spanish have very similar grammar. Thus, the problem arising from different grammatical structure could lead a poor generaliza- tion when training SMT system uses such data. This is one aspect. Another aspect, there still exists some problem even segmenting language to generate sim- ilar vocabulary. In our view, we suppose that simi- lar units should have a proper size. If similar units are too general, it will cause that size of model be- come too large and a over-fitting problem in model itself. Namely, too general similar units could not solve this problem indeed. Too general similar units problem also appears in (Virpioja et al., 2007) where they perform monolingual segmentation at the mor- phological level for Finnish-English translation and put the segmented data to a phrase-based statistical machine translation system. That paper indicates the segmented corpus has lower out-of-vocabulary rates and generates more refined phrases with better gen- eralization ability. However, the results of experi- ment show that they could not improve translation accuracy. In their method, the sentences already had similar units by morphological level segmentation. However, as we mentioned earlier, over-general sim- ilar units also go against on improving the transla- tion quality. On account of those problem, we suppose that data should be segmented through more proper method which …

Content truncated to 500 words