Vocabulary Learning via Optimal Transport for Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7361–7373 August 1–6, 2021. ©2021 Association for Computational Linguistics 7361 Vocabulary Learning via Optimal Transport for Neural Machine Translation Jingjing Xu1, Hao Zhou1, Chun Gan1,2†, Zaixiang Zheng1,3†, Lei Li1 1ByteDance AI Lab 2Math Department, University of Wisconsin–Madison 3Nanjing University {xujingjing.melody,zhouhao.nlp,lileilab}@bytedance.com cgan5@wisc.edu zhengzx@smail.nju.edu.cn Abstract The choice of token vocabulary affects the per- formance of machine translation. This paper aims to figure out what is a good vocabulary and whether one can find the optimal vocab- ulary without trial training. To answer these questions, we first provide an alternative un- derstanding of the role of vocabulary from the perspective of information theory. Motivated by this, we formulate the quest of vocabular- ization – finding the best token dictionary with a proper size – as an optimal transport (OT) problem. We propose VOLT, a simple and efficient solution without trial training. Em- pirical results show that VOLT outperforms widely-used vocabularies in diverse scenar- ios, including WMT-14 English-German and TED’s 52 translation directions. For example, VOLT achieves 70% vocabulary size reduc- tion and 0.5 BLEU gain on English-German translation. Also, compared to BPE-search, VOLT reduces the search time from 384 GPU hours to 30 GPU hours on English-German translation. Codes are available at https: //github.com/Jingjing-NLP/VOLT. 1 Introduction Due to the discreteness of text, vocabulary con- struction ( vocabularization for short) is a prereq- uisite for neural machine translation (NMT) and many other natural language processing (NLP) tasks using neural networks (Mikolov et al., 2013; Vaswani et al., 2017; Gehrmann et al., 2018; Zhang et al., 2018; Devlin et al., 2019). Cur- rently, sub-word approaches like Byte-Pair En- coding (BPE) are widely used in the commu- nity (Ott et al., 2018; Ding et al., 2019; Liu et al., 2020), and achieve quite promising results in prac- tice (Sennrich et al., 2016; Costa-juss`a and Fonol- losa, 2016; Lee et al., 2017; Kudo and Richardson, †This work is done during the internship at ByteDance AI Lab. 2018; Al-Rfou et al., 2019; Wang et al., 2020). The key idea of these approaches is selecting the most frequent sub-words (or word pieces with higher probabilities) as the vocabulary tokens. In information theory, these frequency-based ap- proaches are simple forms of data compression to reduce entropy (Gage, 1994), which makes the re- sulting corpus easy to learn and predict (Martin and England, 2011; Bentz and Alikaniotis, 2016). However, the effects of vocabulary size are not sufficiently taken into account since current ap- proaches only consider frequency (or entropy) as the main criteria. Many previous studies (Sennrich and Zhang, 2019; Ding et al., 2019; Provilkov et al., 2020; Salesky et al., 2020) show that vocab- ulary size also affects downstream performances, especially on low-resource tasks. Due to the lack of appropriate inductive bias about size, trial train- ing (namely traversing all possible sizes) is usually required to search for the optimal size, which takes high computation costs. For …

Content truncated to 500 words