Neural Machine Translation via Binary Code Prediction

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 850–860 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1079 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 850–860 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1079 Neural Machine Translation via Binary Code Prediction Yusuke Oda† Philip Arthur† Graham Neubig‡† Koichiro Yoshino†§ Satoshi Nakamura† † Nara Institute of Science and Technoloty, 8916-5 Takayama-cho, Ikoma, Nara 630-0192, Japan ‡ Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA § Japan Science and Technology Agency, 4-1-8 Hon-machi, Kawaguchi, Saitama 332-0012, Japan {oda.yusuke.on9, philip.arthur.om0}@is.naist.jp, gneubig@cs.cmu.edu, {koichiro, s-nakamura}@is.naist.jp Abstract In this paper, we propose a new method for calculating the output layer in neural machine translation systems. The method is based on predicting a binary code for each word and can reduce computation time/memory requirements of the output layer to be logarithmic in vocabulary size in the best case. In addition, we also intro- duce two advanced approaches to improve the robustness of the proposed model: us- ing error-correcting codes and combining softmax and binary codes. Experiments on two English ↔ Japanese bidirectional translation tasks show proposed models achieve BLEU scores that approach the softmax, while reducing memory usage to the order of less than 1/10 and improving decoding speed on CPUs by x5 to x10. 1 Introduction When handling broad or open domains, machine translation systems usually have to handle a large vocabulary as their inputs and outputs. This is par- ticularly a problem in neural machine translation (NMT) models (Sutskever et al., 2014), such as the attention-based models (Bahdanau et al., 2014; Luong et al., 2015) shown in Figure 1. In these models, the output layer is required to generate a specific word from an internal vector, and a large vocabulary size tends to require a large amount of computation to predict each of the candidate word probabilities. Because this is a significant problem for neural language and translation models, there are a num- ber of methods proposed to resolve this problem, which we detail in Section 2.2. However, none of these previous methods simultaneously satisfies the following desiderata, all of which, we argue, are desirable for practical use in NMT systems: Figure 1: Encoder-decoder-attention NMT model and computation amount of the output layer. Memory efficiency: The method should not re- quire large memory to store the parameters and calculated vectors to maintain scalability in resource-constrained environments. Time efficiency: The method should be able to train the parameters efficiently, and possible to perform decoding efficiently with choos- ing the candidate words from the full proba- bility distribution. In particular, the method should be performed fast on general CPUs to suppress physical costs of computational re- sources for actual production systems. Compatibility with parallel computation: It should be easy for the method to be mini- batched and optimized to run efficiently on GPUs, which are essential for training large NMT models. In this …

Content truncated to 500 words