Transformer with Syntactic Position Encoding for Machine Translation

Proceedings of Recent Advances in Natural Language Processing, pages 1536–1544 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_172 1536 Transformer with Syntactic Position Encoding for Machine Translation Yikuan Xie1, Wenyong Wang1, Mingqian Du2, and Qing He3 1University of Electronic Science and Technology of China 2The Second Research Institute of CAAC 3The Third People’s Hospital of Chengdu City xieyikuan@std.uestc.edu.cn, wangwy@uestc.edu.cn dumingqian@caacetc.com, 123hq@163.com Abstract It has been widely recognized that syntax in- formation can help end-to-end neural machine translation (NMT) systems to achieve better translation. In order to integrate dependency information into Transformer based NMT, ex- isting approaches either exploit words’ lo- cal head-dependent relations, ignoring their non-local neighbors carrying important con- text; or approximate two words’ syntactic re- lation by their relative distance on the de- pendency tree, sacrificing exactness. To ad- dress these issues, we propose global po- sitional encoding for dependency tree, new scheme that facilitates syntactic rela-a tion modeling between any two words with keeping exactness and without immediate neighbor constraint. Experiment results on NC11 German→English, English→German and WMT English→German datasets show that our approach is more effective than the above two strategies. In addition, our experi- ments quantitatively show that compared with higher layers, lower layers of the model are more proper places to incorporate syntax infor- mation in terms of each layer’s preference to the syntactic pattern and the final performance. 1 Introduction Over the past few years, end-to-end neural machine translation (NMT) has shown remarkable progress, achieving promising results on various machine translation tasks. Although NMT models can per- form well even only trained on parallel corpus, they have been found to still benefit from external lin- guistic knowledge (e.g., lexical analysis and pars- ing) (Sennrich and Haddow, 2016), and the same finding was also observed in other tasks (Strubell et al., 2018; Zhang et al., 2019). Therefore, several studies have started to to investigate how to incor- porate dependency information into NMT based on Transformer, which is currently the most advanced end-to-end machine translation backbone. There exist two broad categories of approaches to integrating a sentence’s dependency structure into the Transformer. The first category of ap- proaches (Bugliarello and Okazaki, 2020; Strubell et al., 2018; Bastings et al., 2017) takes the depen- dency tree as a general graph structure data. A graph neural network or a sublayer with a similar mechanism was introduced at the bottom of the Transformer’s encoder to represent and encode the dependency relations among the words in the sen- tence. In this paradigm, however, a word focuses only on its immediate neighbors and ignores nodes that are multi-hop away from it, which may carry important context that helps disambiguate and en- rich the current word’s representation in terms of syntactic structure. In contrast, methods from the second category seek to model two words’ syntactic relation rely- ing on their distance from the dependency tree. The distance can be two words’ relative depth on the tree or the length of the path between the two nodes (Wang et al., 2019a; Omote et al., 2019). This …

Content truncated to 500 words