Double Path Networks for Sequence to Sequence Learning

Proceedings of the 27th International Conference on Computational Linguistics, pages 3064–3074 Santa Fe, New Mexico, USA, August 20-26, 2018. 3064 Double Path Networks for Sequence to Sequence Learning 1Kaitao Song, 2Xu Tan, 3Di He, 1Jianfeng Lu, 2Tao Qin and 2Tie-Yan Liu 1Nanjing University of Science and Technology 2Microsoft Research 3Key Laboratory of Machine Perception, MOE, School of EECS, Peking University {kt.song, lujf}@njust.edu.cn {xuta, taoqin, tyliu}@microsoft.com {di he}@pku.edu.cn Abstract Encoder-decoder based Sequence to Sequence learning (S2S) has made remarkable progress in recent years. Different network architectures have been used in the encoder/decoder. Among them, Convolutional Neural Networks (CNN) and Self Attention Networks (SAN) are the promi- nent ones. The two architectures achieve similar performances but use very different ways to encode and decode context: CNN use convolutional layers to focus on the local connectivity of the sequence, while SAN uses self-attention layers to focus on global semantics. In this work we propose Double Path Networks for Sequence to Sequence learning (DPN-S2S), which leverage the advantages of both models by using double path information fusion. During the encoding step, we develop a double path architecture to maintain the information coming from different paths with convolutional layers and self-attention layers separately. To effectively use the en- coded context, we develop a cross attention module with gating and use it to automatically pick up the information needed during the decoding step. By deeply integrating the two paths with cross attention, both types of information are combined and well exploited. Experiments show that our proposed method can significantly improve the performance of sequence to sequence learning over state-of-the-art systems. 1 Introduction Sequence to Sequence learning (S2S) (Cho et al., 2014; Sutskever et al., 2014) is one of the challenging tasks in artificial intelligence, which covers machine translation, document summarization and question answering, etc. It has attracted more and more attention in recent years due to the success of deep learn- ing. Sequence to sequence learning is usually developed based on a general encoder-decoder framework: The encoder reads the source sequence and generates a set of representations. After that, the decoder es- timates the conditional probability of each target token given the source representations and its preceding tokens. Different network architectures for sequence to sequence modelling have been designed based on this framework. Long Short-Term Memory (LSTM) based model (Cho et al., 2014) is the most popular. In the LSTM based model, the tokens are encoded/decoded using LSTM units, which can effectively summarize the temporal information of the sentences in source/target domains. However, because of its recurrent nature, the LSTM based model is difficult to parallel, so the training efficiency becomes the major challenge. Recently, Convolutional Neural Network (CNN) based (Gehring et al., 2017a) and Self Attention Network (SAN) based (Vaswani et al., 2017) models have been proposed, which achieved significant improvements in accuracy. Both models replace the recurrent structure with networks that are easy to parallelize, leading to training time acceleration. CNN based and SAN based sequence to sequence models encode and decode information in …

Content truncated to 500 words