MonoByte: A Pool of Monolingual Byte-level Language Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 378–388 October 12–17, 2022. 378 Incorporating Causal Analysis into Diversified and Logical Response Generation Jiayi Liu1, Wei Wei2,3∗, Zhixuan Chu4, Xing Gao1, Ji Zhang1, Tan Yan4, Yulin Kang4 1Alibaba Group, China 2CCIIP Laboratory, Huazhong University of Science and Technology, China 3Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL), China 4Ant Group, China {ljy269999,chuzhixuan.czx,gaoxing.gx,zj122146}@alibaba-inc.com weiw@hust.edu.cn, tanyan.ty@gmail.com, yulin.kyl@antgroup.com Abstract Although the Conditional Variational Auto- Encoder (CVAE) model can generate more di- versified responses than the traditional Seq2Seq model, the responses often have low relevance with the input words or are illogical with the question. A causal analysis is carried out to study the reasons behind, and a methodol- ogy of searching for the mediators and mit- igating the confounding bias in dialogues is provided. Specifically, we propose to predict the mediators to preserve relevant information and auto-regressively incorporate the mediators into generating process. Besides, a dynamic topic graph guided conditional variational auto- encoder (TGG-CVAE) model is utilized to com- plement the semantic space and reduce the con- founding bias in responses. Extensive experi- ments demonstrate that the proposed model is able to generate both relevant and informative responses, and outperforms the state-of-the-art in terms of automatic metrics and human eval- uations. 1 Introduction With recent advances in deep learning and read- ily available large-scale dialogue data, generation- based methods have become one of the most pre- vailing methods for building dialogue systems. Based on the Seq2seq framework (Sutskever et al., 2014; Cho et al., 2014), generation-based models learn to map the input post to its corresponding response through an encoding-decoding strategy and are trained in end-to-end manners (Shang et al., 2015; Sordoni et al., 2015; Vinyals and Le, 2015). However, Seq2seq model tends to produce generic and safe responses (Li et al., 2015) such as “So am I” or “I don’t know”. Researchers conjecture that the cause of this phenomenon is that one certain post can be replied by multiple responses (i.e., one- to-many mapping), and the maximum likelihood estimation (MLE) training would average out these ∗*Corresponding author. Post: Have you had dinner? Response1: Yeah, sure! Response2: Yes, I had it at McDonald’s. Response3: Nope, I’m busy with my work. Response4: Yes, I’ve had it. I tried a nearby restaurant that features Thai food. Table 1: An illustration of a general question and its mul- tiple valid responses. The direct responding semantics (marked in red) are semantically homogeneous because they have to reply the issue directly. The supplementary semantics are more diversified because they add more information to explain or supplement the corresponding direct responding semantics. responses and produce a more bland and generic candidate. To tackle this problem and model the one-to- many mapping relationships in dialogues, (Zhao et al., 2017) firstly leverages Conditional Varia- tional Auto-Encoder (CVAE) model to map the input post into a semantic distribution, instead of a fixed vector as used in the vanilla Seq2seq model. The decoder then decodes the sampled points from …

Content truncated to 500 words