FNet: Mixing Tokens with Fourier Transforms

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 407 - 418 July 10-15, 2022 ©2022 Association for Computational Linguistics Diversifying Neural Dialogue Generation via Negative Distillation Yiwei Li, Shaoxiong Feng, Bin Sun, Kan Li School of Computer Science, Beijing Institute of Technology {liyiwei,shaoxiongfeng,binsun,likan}@bit.edu.cn Abstract Generative dialogue models suffer badly from the generic response problem, limiting their applications to a few toy scenarios. Recently, an interesting approach, namely negative train- ing, has been proposed to alleviate this prob- lem by reminding the model not to gener- ate high-frequency responses during training. However, its performance is hindered by two issues, ignoring low-frequency but generic re- sponses and bringing low-frequency but mean- ingless responses. In this paper, we propose a novel negative training paradigm, called nega- tive distillation, to keep the model away from the undesirable generic responses while avoid- ing the above problems. First, we introduce a negative teacher model that can produce query- wise generic responses, and then the student model is required to maximize the distance with multi-level negative knowledge. Em- pirical results show that our method outper- forms previous negative training methods sig- nificantly.1 1 Introduction In the past few years, data-driven response gener- ation (Vinyals and Le, 2015; Shang et al., 2015; Vougiouklis et al., 2016) has achieved impressive performance, drawing continuously increasing at- tention from academia and industry. Convention- ally, with the guidance of maximum likelihood estimation (MLE), neural dialogue models are ex- pected to maximize the probability of generating the corresponding reference given any query. Un- fortunately, due to the many-to-one phenomenon (see Table 1), a characteristic of the dialogue task (Csáky et al., 2019), these models are prone to pro- duce safe but generic responses (e.g., I don’t know (Li et al., 2016)), which sets an obstacle for the generative dialogue system to be deployed widely. Some researchers tried to redesign the objective 1The code and preprocessed data are available at https://github.com/Yiwei98/dialogue-negative-distillation. of models to meet the requirement of diverse re- sponses instead of MLE, such as MMI (Li et al., 2016), AdaLabel (Wang et al., 2021), and IAT (Zhou et al., 2021). Besides, several studies (Ku- likov et al., 2019; Holtzman et al., 2020) proposed more advanced decoding strategies to alleviate the problem of generic responses. Indeed, the above methods boost the diversity of responses by remind- ing the model what should be said. However, inspired by negative training (Kim et al., 2019; Ma et al., 2021), we argue that it is also necessary to tell the dialogue model what not to say. To alleviate the problem of generic responses, He and Glass (2020) negatively updates the parameters when identifying the high-frequency responses. Li et al. (2020a) punishes the behaviors of generating repetitive or high-frequency tokens by using the unlikelihood objective (Welleck et al., 2020). Although the negative-training based methods enhance the diversity of responses, there still exists two drawbacks: First, they regard high-frequency tokens or utterances as negative candidates. How- ever, …

Content truncated to 500 words