Generalized Supervised Attention for Text Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4991–5003 August 1–6, 2021. ©2021 Association for Computational Linguistics 4991 Generalized Supervised Attention for Text Generation∗ Yixian Liu1, Liwen Zhang2, Xinyu Zhang3, Yong Jiang, Yue Zhang4, Kewei Tu2 1Interactive Entertainment Group, Tencent 2School of Information Science and Technology, ShanghaiTech University 3Language Technologies Institute, Carnegie Mellon University, USA 4School of Engineering, Westlake Universtiy 4Institute of Advanced Technology, Westlake Institute for Advanced Study easingliu@tencent.com xinyuzh2@cs.cmu.edu {zhanglw1, tukw}@shanghaitech.edu.cn zhangyue@westlake.edu.cn Abstract The attention-based encoder-decoder frame- work is widely used in many natural lan- guage generation tasks. The attention mecha- nism builds alignments between target words and source items that facilitate text genera- tion. Previous work proposes supervised at- tention that uses human knowledge to guide the attention mechanism to learn better align- ments. However, well-designed supervision built from ideal alignments can be costly or even infeasible. In this paper, we build a Gen- eralized Supervised Attention method (GSA) based on quasi alignments, which specify can- didate sets of alignments and are much eas- ier to obtain than ideal alignments. We design a Summation Cross-Entropy (SCE) loss and a Supervised Multiple Attention (SMA) struc- ture to accommodate quasi alignments. Exper- iments on three text generation tasks demon- strate that GSA improves generation perfor- mance and is robust against errors in attention supervision. 1 Introduction The encoder-decoder framework has been applied to various natural language generation (NLG) tasks, such as neural machine translation (Cho et al., 2014; Luong et al., 2015; Vaswani et al., 2017; Liu et al., 2016), text generation (Wiseman et al., 2017; Puduppully et al., 2019), text summarization (Liu and Lapata, 2019; Lin et al., 2018), image cap- tioning (Anderson et al., 2018), dialogue systems (Liu et al., 2018), and so on. The attention mech- anism (Bahdanau et al., 2015) plays a significant role in the framework, which automatically extracts the alignments between the target and the source for predicting the next target output. One disadvan- tage of the vanilla attention mechanisms is that the ∗The work was done when Yixian Liu and Xinyu Zhang were students of Shanghaitech University. Kewei Tu is the corresponding author. automatic weights do not necessarily encode prior knowledge, such as the alignments between input and output (Jain and Wallace, 2019). To alleviate this problem, supervised attention was considered (Liu et al., 2016; Mi et al., 2016; Kamigaito et al., 2017; Nguyen et al., 2018; Nguyen and Nguyen, 2018), which shows that human knowledge is help- ful for guiding the learning process of attention models. Previous work on supervised attention assumes access to ideal alignments. Unfortunately, obtain- ing ideal alignments is infeasible or extremely costly, for most NLG tasks. For example in Fig- ure 1, for the AMR-to-text generation task, given the AMR graph for sentence “From among them, pick out 50 for submission to an assessment com- mittee to assess.”, the ideal alignment of the last word “assess” is node (10). As the names of (8) and (10) are the same, it is not easy to pick …

Content truncated to 500 words