Shared-Private Bilingual Word Embeddings for Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3613–3622 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3613 Shared-Private Bilingual Word Embeddings for Neural Machine Translation Xuebo Liu† Derek F. Wong†∗ Yang Liu‡ Lidia S. Chao† Tong Xiao§ Jingbo Zhu§ †NLP2CT Lab / Department of Computer and Information Science, University of Macau, Macau ‡Department of Computer Science and Technology, Tsinghua University, Beijing, China §Northeastern University, Shenyang, China nlp2ct.xuebo@gmail.com, {derekfw,lidiasc}@um.edu.mo, liuyang2011@tsinghua.edu.cn, {xiaotong,zhujingbo}@mail.neu.edu.cn Abstract Word embedding is central to neural machine translation (NMT), which has attracted inten- sive research interest in recent years. In NMT, the source embedding plays the role of the entrance while the target embedding acts as the terminal. These layers occupy most of the model parameters for representation learn- ing. Furthermore, they indirectly interface via a soft-attention mechanism, which makes them comparatively isolated. In this paper, we propose shared-private bilingual word em- beddings, which give a closer relationship be- tween the source and target embeddings, and which also reduce the number of model pa- rameters. For similar source and target words, their embeddings tend to share a part of the features and they cooperatively learn these common representation units. Experiments on 5 language pairs belonging to 6 different lan- guage families and written in 5 different al- phabets demonstrate that the proposed model provides a significant performance boost over the strong baselines with dramatically fewer model parameters. 1 Introduction With the introduction of ever more powerful ar- chitectures, neural machine translation (NMT) has become the most promising machine transla- tion method (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015). For word representation, different architectures— including, but not limited to, recurrence-based (Chen et al., 2018), convolution-based (Gehring et al., 2017) and transformation-based (Vaswani et al., 2017) NMT models—have been taking ad- vantage of the distributed word embeddings to capture the syntactic and semantic properties of words (Turian et al., 2010). ∗Corresponding author Long Rd Lange Rd (a) Standard Long Rd Lange Rd (b) Shared-private Figure 1: Comparison between (a) standard word em- beddings and (b) shared-private word embeddings. In (a), the English word “Long” and the German word “Lange”, which have similar lexical meanings, are rep- resented by two private d-dimension vectors. While in (b), the two word embeddings are made up of two parts, indicating the shared (lined nodes) and the private (un- lined nodes) features. This enables the two words to make use of common representation units, leading to a closer relationship between them. NMT usually utilizes three matrices to rep- resent source embeddings, target input embed- dings, and target output embeddings (also known as pre-softmax weight), respectively. These em- beddings occupy most of the model parameters, which constrains the improvements of NMT be- cause the recent methods become increasingly memory-hungry (Vaswani et al., 2017; Chen et al., 2018).1 Even though converting words into sub- word units (Sennrich et al., 2016b), nearly 55% of model parameters are used for word representation in the …

Content truncated to 500 words