How Transformer Revitalizes Character-based Neural Machine Translation: An Investigation on Japanese-Vietnamese Translation Systems

How Transformer Revitalizes Character-based Neural Machine Translation: An Investigation on Japanese-Vietnamese Translation Systems Thi-Vinh Ngo1, Thanh-Le Ha2, Phuong-Thai Nguyen3, Le-Minh Nguyen4 1University of Information and Communication Technology, TNU, Vietnam 2Institute of Anthropomatics and Robotics, KIT, Germany 3University of Engineering and Technology, VNU, Vietnam 4School of Information Science, JAIST, Japan ntvinh@ictu.edu.vn,thanh-le.ha@kit.edu,thainp@vnu.edu.vn,nguyenml@jaist.ac.jp Abstract While translating between East Asian languages, many works have discovered clear advantages of using characters as the translation unit. Unfortunately, traditional recurrent neural machine translation systems hinder the practical us- age of those character-based systems due to their architec- tural limitations. They are unfavorable in handling extremely long sequences as well as highly restricted in parallelizing the computations. In this paper, we demonstrate that the new transformer architecture can perform character-based trans- lation better than the recurrent one. We conduct experiments on a low-resource language pair: Japanese-Vietnamese. Our models considerably outperform the state-of-the-art systems which employ word-based recurrent architectures. 1. Introduction Neural machine translation (NMT) has achieved the state- of-the-art performances in recent machine translation cam- paigns for many language pairs due to its fluent and accu- rate outputs [1–3]. Yet often those outputs from NMT are over-translated, where words or phrases are redundantly re- peated in the translations, thus affecting their readability. One reason leading to over-translation in NMT is, unlike the traditional statistical machine translation (SMT), basic NMT architectures and their beam search do not explicitly model coverage. Furthermore, their content-based attention mechanism, which effectively aligns an arbitrary number of source words to the only target word at a time when de- coding without any length control, intensifies the problem. Consequently, the best and simplest strategy to avoid over- translation when translating between two languages having lots of length mismatches might be to segment the texts in some way so that the systems could work with one-to-one alignments as many as possible1. On the other hand, given sufficiently large data, a well-designed NMT architecture is capable of automatically learning good alignments with its attention mechanism. 1By terming one-to-one alignment, we mean that one translation unit of the source sentence corresponds to one translation unit of the target sentence and vice versa. For translation systems between Romance, Balto-Slavic and Germanic languages, unsupervised subword segmenta- tion methods, such as Byte-Pair Encoding [4], are often used and they show great improvements. Subword-based transla- tion systems possess two important advantages. First, they reduce the vocabulary size, hence, making the model mem- ory and performance-efficient. Second, they are able to com- petently deal with unknown and rare words. For many other languages in East, South and South-East Asia, however, the preprocessing frequently requires more complicated and ex- pensive supervised tokenization methods to segment the texts into decent translation units, since in those languages, word boundaries are not signaled by the whitespaces. In order to liberate the whole translation system from the dependencies on those non-trivial tokenization methods (in- cluding subword segmentation) while still effectively deal- ing with the out-of-vocabulary (OOV) problem, character- level approaches2 have been investigated. However, those approaches have not …

Content truncated to 500 words