Multi-Unit Transformers for Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1047–1059, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1047 Multi-Unit Transformers for Neural Machine Translation Jianhao Yan Fandong Meng Jie Zhou Pattern Recognition Center, WeChat AI, Tencent Inc, Beijing, China {elliottyan,fandongmeng,withtomzhou}@tencent.com Abstract Transformer models (Vaswani et al., 2017) achieve remarkable success in Neural Ma- chine Translation. Many efforts have been de- voted to deepening the Transformer by stack- ing several units (i.e., a combination of Multi- head Attentions and FFN) in a cascade, while the investigation over multiple parallel units draws little attention. In this paper, we pro- pose the Multi-Unit TransformErs (MUTE), which aim to promote the expressiveness of the Transformer by introducing diverse and complementary units. Specifically, we use sev- eral parallel units and show that modeling with multiple units improves model performance and introduces diversity. Further, to better leverage the advantage of the multi-unit set- ting, we design biased module and sequen- tial dependency that guide and encourage com- plementariness among different units. Exper- imental results on three machine translation tasks, the NIST Chinese-to-English, WMT’14 English-to-German and WMT’18 Chinese-to- English, show that the MUTE models signif- icantly outperform the Transformer-Base, by up to +1.52, +1.90 and +1.10 BLEU points, with only a mild drop in inference speed (about 3.1%). In addition, our methods also surpass the Transformer-Big model, with only 54% of its parameters. These results demonstrate the effectiveness of the MUTE, as well as its ef- ficiency in both the inference process and pa- rameter usage. 1 1 Introduction Transformer based models (Vaswani et al., 2017) have been proven to be very effective in build- ing the state-of-the-art Neural Machine Transla- tion (NMT) systems via neural networks and atten- tion mechanism (Sutskever et al., 2014; Bahdanau 1Code is available at https://github.com/Ellio ttYan/Multi Unit Transformer et al., 2014). Following the standard Sequence-to- Sequence architecture, Transformer models consist of two essential components, namely the encoder and decoder, which rely on stacking several identi- cal layers, i.e., multihead attentions and position- wise feed-forward network. Multihead attentions and position-wise feed- forward network, together as a basic unit, plays an essential role in the success of Transformer mod- els. Some researchers (Bapna et al., 2018; Wang et al., 2019a) propose to improve the model capac- ity by stacking this basic unit many times, i.e., deep Transformers, and achieve promising results. Nev- ertheless, as an orthogonal direction, investigation over multiple parallel units draws little attention. Compared with single unit models, multiple par- allel unit layout is more expressive to capture com- plex information flow (Tao et al.; Meng et al., 2019; Li et al., 2018, 2019) in two aspects. First, this multiple-unit layout boosts the model by its varied feature space composition and different attentions over inputs. With this diversity, multi-unit models advance in expressiveness. Second, for the multi- unit setting, one unit could mitigate the deficiency of other units and compose a more expressive net- work, in a complementary way. In this paper, we propose the Multi-Unit …

Content truncated to 500 words