Lattice-Based Transformer Encoder for Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3090–3097 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3090 Lattice-Based Transformer Encoder for Neural Machine Translation Fengshun Xiao1,2, Jiangtong Li2,3, Hai Zhao1,2,∗, Rui Wang4, Kehai Chen4 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China 3College of Zhiyuan, Shanghai Jiao Tong University, China 4National Institute of Information and Communications Technology (NICT) {felixxiao, keep moving-lee}@sjtu.edu.cn, zhaohai@cs.sjtu.edu.cn, {wangrui, khchen}@nict.go.jp Abstract Neural machine translation (NMT) takes deterministic sequences for source representations. However, either word- level or subword-level segmentations have multiple choices to split a source sequence with different word segmentors or different subword vocabulary sizes. We hypothesize that the diversity in segmentations may affect the NMT performance. To integrate different segmentations with the state-of-the-art NMT model, Transformer, we propose lattice-based encoders to explore effective word or subword representation in an automatic way during training. We propose two methods: 1) lattice positional encoding and 2) lattice-aware self-attention. These two methods can be used together and show complementary to each other to further improve translation performance. Experiment results show superiorities of lattice-based encoders word-level and subword-level representationsin over conventional Transformer encoder. 1 Introduction Neural machine translation (NMT) has achieved great progress with the evolvement of model structures under an encoder-decoder framework (Sutskever et al., 2014; Bahdanau et al., 2014). Recently, the self-attention based Transformer model has achieved state-of-the- art performance on multiple language pairs (Vaswani et al., 2017; Marie et al., 2018). Both representations of source and target sentences in ∗ Corresponding author. This paper was partially supported by National Key Research and Development Program of China (No. 2017YFB0304100) and key projects of National Natural Science Foundation of China (No. U1836222 and No. 61733011). Rui Wang was partially supported by JSPS grant-in-aid for early-career scientists (19K20354): “Unsupervised Neural Machine Translation in Universal Scenarios” and NICT tenure-track researcher startup fund “Toward Intelligent Machine Translation”. v8 mao yi fa zhan ju fu zong cai v7 v6 v5 v4 v3 v2 v1 v0 mao-yi fa-zhan ju fu zong-cai fa-zhan-ju mao-yi-fa-zhan fu-zong-cai v0 v1 v2 v3 mao-yi-fa-zhan ju fu-zong-cai v0 v1 v2 v3 mao-yi fa-zhan-ju fu-zong-cai v0 v1 v2 v3 v4 v5 mao-yi fa-zhan ju fu zong-cai (1)Segmentaion 1 (2)Segmentaion 2 (3)Segmentation 3 (4)Lattice v8 mao yi fa zhan ju fu zong cai v7 v6 v5 v4 v3 v2 v1 v0 e0:2:mao-yi e2:4:fa-zhan e4:5:ju e5:6:fu e6:8:zong-cai e2:5:fa-zhan-ju e0:4:mao-yi-fa-zhan e5:8:fu-zong-cai c1 c2 c3 c4 c5 c6 c7 c8 Figure 1: Incorporating three different segmentation for a lattice graph. The original sentence is “mao-yi- fa-zhan-ju-fu-zong-cai”. In Chinese it is “贸易发展局 副总裁”. In English it means “The vice president of Trade Development Council” NMT can be factorized in character (Costa-Jussa and Fonollosa, 2016), word (Sutskever et al., 2014), or subword (Sennrich et al., 2015) level. However, only using 1-best segmentation as inputs limits NMT encoders to express source sequences sufficiently and reliably. Many East …

Content truncated to 500 words