Self-Attention with Cross-Lingual Position Representation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1679–1685 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1679 Self-Attention with Cross-Lingual Position Representation Liang Ding† Longyue Wang‡ Dacheng Tao† †UBTECH Sydney AI Centre, School of Computer Science, Faculty of Engineering, The University of Sydney {ldin3097,dacheng.tao}@sydney.edu.au ‡Tencent AI Lab vinnylywang@tencent.com Abstract Position encoding (PE), an essential part of self-attention networks (SANs), is used to pre- serve the word order information for natural language processing tasks, generating fixed position indices for input sequences. How- ever, in cross-lingual scenarios, e.g., machine translation, the PEs of source and target sen- tences are modeled independently. Due to word order divergences in different languages, modeling the cross-lingual positional relation- ships might help SANs tackle this problem. In this paper, we augment SANs with cross- lingual position representations to model the bilingually aware latent structure for the in- put sentence. Specifically, we utilize brack- eting transduction grammar (BTG)-based re- ordering information to encourage SANs to learn bilingual diagonal alignments. Experi- mental results on WMT’14 English⇒German, WAT’17 Japanese⇒English, and WMT’17 Chinese⇔English translation tasks demon- strate that our approach significantly and con- sistently improves translation quality over strong baselines. Extensive analyses confirm that the performance gains come from the cross-lingual information. 1 Introduction Although self-attention networks (SANs) (Lin et al., 2017) have achieved the state-of-the-art per- formance on several natural language processing (NLP) tasks (Vaswani et al., 2017; Devlin et al., 2019; Radford et al., 2018), they possess the in- nate disadvantage of sequential modeling due to the lack of positional information. Therefore, abso- lute position encoding (APE) (Vaswani et al., 2017) and relative position encoding (RPE) (Shaw et al., 2018) were introduced to better capture the sequen- tial dependencies. However, either absolute or rela- tive PE is language-independent and its embedding Bush with Sharon held a talk Bush held a talk with Sharon [source] [re-ordered] 0 1 2 3 4 5 [abs POS] 0 3 4 5 1 2 [XL POS] 布什 与 沙龙 举行 了 会谈 [target] 布什 与 沙龙 举行 了 会谈 Bush held a talk with Sharon (a) BTG tree based cross-lingual structure for En-Zh Inverted Straight (b) Absolute(abs) Position vs. Cross-Lingual(XL) Position Figure 1: Illustration of cross-lingual position for English⇒Chinese translation task. (a) BTG tree shows the cross-lingual preordering. The top-left corner is the transduction grammar. (b) the difference between abso- lute position encoding (APE) and our proposed cross- lingual position encoding (XL PE) . remains fixed. This inhibits the capacity of SANs when modelling multiple languages, which have diverse word orders and structures (Gell-Mann and Ruhlen, 2011). Recent work have shown that mod- eling cross-lingual information (e.g., alignment or reordering) at encoder or attention level improves translation performance for different language pairs (Cohn et al., 2016; Du and Way, 2017; Zhao et al., 2018; Kawara et al., 2018). Inspired by their work, we propose to augment SANs with cross-lingual representations, by en- coding reordering indices at embedding level. Tak- ing English⇒Chinese translation task for exam- ple, …

Content truncated to 500 words