Local Byte Fusion for Neural Machine Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7199–7214 July 9-14, 2023 ©2023 Association for Computational Linguistics Local Byte Fusion for Neural Machine Translation Makesh Narsimhan Sreedhar1, Xiangpeng Wan2, Yu Cheng3, Junjie Hu1 1University of Wisconsin-Madison, 2NetMind.AI and ProtagoLabs, 3Microsoft Research {msreedhar,junjie.hu}@wisc.edu Abstract Subword tokenization schemes are the dom- inant technique used in current NLP models. However, such schemes can be rigid and tok- enizers built on one corpus may not adapt well to other parallel corpora. It has also been ob- served that in multilingual corpora, subword to- kenization schemes oversegment low-resource languages, leading to a drop in translation per- formance. An alternative to subword tokeniz- ers is byte-based tokenization, i.e., tokeniza- tion into byte sequences using the UTF-8 en- coding scheme. Byte tokens often represent inputs at a sub-character granularity, i.e., one character can be represented by a span of byte tokens. This results in much longer byte se- quences that are hard to interpret without ag- gregating local information from multiple byte tokens. In this paper, we propose a Local Byte Fusion (LOBEF) method for byte-based ma- chine translation—utilizing byte n-gram and word boundaries—to aggregate local semantic information. Extensive experiments on mul- tilingual translation, zero-shot cross-lingual transfer, and domain adaptation reveal a con- sistent improvement over vanilla byte-based models. Further analysis also indicates that our byte-based models are parameter-efficient and perform competitive to subword models. 1 Introduction Multilingual neural machine translation (NMT) has proven effective to transfer knowledge learned from a high-resource language to a low-resource language. However, existing multilingual NMT models still rely on a pre-built subword tok- enizer (e.g., BPE (Sennrich et al., 2016), Senten- cePiece (Kudo and Richardson, 2018)) to tokenize a sentence into a sequence of subword units. This has two drawbacks. First, once the tokenizer is fixed, we lose the flexibility of changing the word tokenization if we aim to fine-tune the NMT model on another parallel corpus of interest for adaptation. Second, when a subword tokenizer is built on unbal- anced multilingual data, word tokens from a low- resource language are usually under-represented, resulting in over-segmentation of a word into many single characters. A recent study (Rust et al., 2021) measures over-segmentation by the fertility score of a subword scheme which indicates how many subwords a whole word is broken down into. It shows a negative correlation between the fertility score and the performance of multilingual models on the over-segmented languages. Although character-based models have often been proposed as a solution to these problems (Gupta et al., 2019; Libovický and Fraser, 2020; Li et al., 2021), they come with their own tradeoffs re- lated to the significant overhead of processing long character sequences during training and inference (Libovický and Fraser, 2020). Besides, these mod- els still adopt a fixed vocabulary of characters, lead- ing to the same issue as a fixed subword tokenizer for adaptation. Another point of difference is that in character-based methods, the vocabulary still consists of one unique embedding …

Content truncated to 500 words