Focus-Constrained Attention Mechanism for CVAE-based Response Generation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2021–2030 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2021 Focus-Constrained Attention Mechanism for CVAE-based Response Generation Zhi Cui1, Yanran Li2, Jiayi Zhang1, Jianwei Cui1, Chen Wei1, Bin Wang1 1Xiaomi AI Lab 2The Hong Kong Polytechnic University {cuizhi,zhangjiayi3,cuijianwei,weichen,wangbin11}@xiaomi.com csyli@comp.polyu.edu.hk Abstract To model diverse responses for a given post, one promising way is to introduce a latent vari- able into Seq2Seq models. The latent variable is supposed to capture the discourse-level in- formation and encourage the informativeness of target responses. However, such discourse- level information is often too coarse for the decoder to be utilized. To tackle it, our idea is to transform the coarse-grained discourse- level information into fine-grained word-level information. Specifically, we firstly measure the semantic concentration of corresponding target response on the post words by introduc- ing a fine-grained focus signal. Then, we pro- pose a focus-constrained attention mechanism to take full advantage of focus in well aligning the input to the target response. The experi- mental results demonstrate that by exploiting the fine-grained signal, our model can generate more diverse and informative responses com- pared with several state-of-the-art models.1 1 Introduction Nowadays, developing intelligent open-domain conversational systems has become an active re- search field (Perez-Marin and Pascual-Nieto, 2011; Shum et al., 2018). Compared with rule-based and retrieval-based methods, neural generative models have attracted increasing attention because they do not need extensive feature engineering and have achieved promising results recently with large- scale conversational data (Vinyals and Le, 2015; Sordoni et al., 2015; Shang et al., 2015). Typically, neural generative models are trained to learn the post-response mappings based on 1https://github.com/cuizhi555/Focus-Constrained- Attention-Mechanism-for-CVAE-based-Response- Generation. the Seq2Seq architecture using maximum likeli- hood (MLE) training objective. This kind of objec- tive induces the model to treat the post-response relationship as one-to-one mappings. However, the conversations in the real world often embodies one-to-many relationships, where a post is often as- sociated with multiple valid responses (Zhou et al., 2017). Due to this discrepancy, standard Seq2Seq models tend to generate high-frequency but trivial responses such as “I don’t know” or “I’m ok” (Li et al., 2016). To address this issue, one promising research line resorts to Conditional Variational Autoencoder (CVAE), which introduces a latent variable to Seq2Seq models through variational learning. The latent variable is supposed to capture the discourse- level semantics of target response and in turn en- courage the response informativeness. Recent lit- erature along this line attempted to improve the model performance by putting extra control on the latent variable (Zhao et al., 2017; Gu et al., 2018; Gao et al., 2019). Despite the control, these meth- ods still relied on the discourse-level latent variable, which is too coarse for the decoders to mine suffi- cient guiding signals at each generation step. As a result, these variational models are observed to ignore the latent variable (Zhao et al., 2017; Gu et al., 2018; Gao et al., 2019) and to generate se- mantically irrelevant or grammatically disfluent responses (Qiu …

Content truncated to 500 words