Generative Bridging Network for Neural Sequence Prediction

Proceedings of NAACL-HLT 2018, pages 1706–1715 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Generative Bridging Network for Neural Sequence Prediction Wenhu Chen,1 Guanlin Li,2 Shuo Ren,5 Shujie Liu,3 Zhirui Zhang,4 Mu Li,3 Ming Zhou3 University of California, Santa Barbara1 Harbin Institute of Technology2 Microsoft Research Asia3 University of Science and Technology of China4 Beijing University of Aeronautics and Astronautics5 wenhuchen@cs.ucsb.edu epsilonlee.green@gmail.com {v-shure, shujliu, v-zhirzh, muli, mingzhou}@microsoft.com Abstract In order to alleviate data sparsity and over- fitting problems in maximum likelihood esti- mation (MLE) for sequence prediction tasks, we propose the Generative Bridging Network (GBN), in which a novel bridge module is in- troduced to assist the training of the sequence prediction model (the generator network). Un- like MLE directly maximizing the conditional likelihood, the bridge extends the point-wise ground truth to a bridge distribution condi- tioned on it, and the generator is optimized to minimize their KL-divergence. Three different GBNs, namely uniform GBN, language-model GBN and coaching GBN, are proposed to pe- nalize confidence, enhance language smooth- ness and relieve learning burden. Experiments conducted on two recognized sequence predic- tion tasks (machine translation and abstractive text summarization) show that our proposed GBNs can yield significant improvements over strong baselines. Furthermore, by analyz- ing samples drawn from different bridges, ex- pected influences on the generator are verified. 1 Introduction Sequence prediction has been widely used in tasks where the outputs are sequentially structured and mutually dependent. Recently, massive explo- rations in this area have been made to solve prac- tical problems, such as machine translation (Bah- danau et al., 2014; Ma et al., 2017; Norouzi et al., 2016), syntactic parsing (Vinyals et al., 2015), spelling correction (Bahdanau et al., 2014), image captioning (Xu et al., 2015) and speech recogni- tion (Chorowski et al., 2015). Armed with mod- ern computation power, deep LSTM (Hochreiter and Schmidhuber, 1997) or GRU (Chung et al., 2014) based neural sequence prediction models have achieved the state-of-the-art performance. The typical training algorithm for sequence prediction is Maximum Likelihood Estimation 𝑥1 𝑥2 𝐴𝑡𝑡 𝑥𝑇 𝑦1 𝑗 𝑦2 𝑗 𝑦𝑇𝑗 𝑗 𝑠 𝑦1 𝑗 𝑦𝑇𝑗−1 𝑗 𝑿 𝒀𝒋 𝑦1 ∗ 𝑦2 ∗ 𝑦𝑇′ ∗ 𝒀∗ 𝒀𝒋 𝑃𝜂(𝑌|𝑌∗) 𝑦1 𝑗 𝑦2 𝑗 𝑦𝑇𝑗 𝑗 𝐵𝑟𝑖𝑑𝑔𝑒 𝑚𝑜𝑑𝑢𝑙𝑒 𝐺𝑒𝑛𝑒𝑟𝑎𝑡𝑜𝑟 𝑛𝑒𝑡𝑤𝑜𝑟𝑘 𝒀𝟏 𝒀𝑲 … … 𝑃𝜃(𝑌|𝑋) 𝐵𝑟𝑖𝑑𝑔𝑒 𝑠𝑎𝑚𝑝𝑙𝑒𝑠 … … … … … … … Figure 1: The overall architecture of our novel Gen- erative Bridging Network (GBN). Two main compo- nents, namely the generator network and the bridge module, are connected through samples (Y 1 . . . Y K in red) from the bridge module during training time. (We sometimes call them generator and bridge in brief re- spectively in the following discussion.) The generator is implemented through an attentive encoder-decoder, where in the figure Att represents the attention module. (MLE), which maximizes the likelihood of the tar- get sequences conditioned on the source ones: ✓⇤ = argmax ✓ E (X,Y ⇤)⇠D log p✓(Y ⇤|X) (1) Despite the popularity of MLE or teacher forc- ing …

Content truncated to 500 words