Vocabulary Pyramid Network: Multi-Pass Encoding and Decoding with Multi-Level Vocabularies for Response Generation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3774–3783 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3774 Vocabulary Pyramid Network: Multi-Pass Encoding and Decoding with Multi-Level Vocabularies for Response Generation Cao Liu1,2, Shizhu He1, Kang Liu1,2, Jun Zhao1,2 1 National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China 2 University of Chinese Academy of Sciences, Beijing, 100049, China {cao.liu, shizhu.he, kliu, jzhao}@nlpr.ia.ac.cn Abstract We study the task of response generation. Conventional methods employ a fixed vocab- ulary and one-pass decoding, which not only make them prone to safe and general respons- es but also lack further refining to the first gen- erated raw sequence. To tackle the above t- wo problems, we present a Vocabulary Pyra- mid Network (VPN) which is able to incor- porate multi-pass encoding and decoding with multi-level vocabularies into response genera- tion. Specifically, the dialogue input and out- put are represented by multi-level vocabular- ies which are obtained from hierarchical clus- tering of raw words. Then, multi-pass encod- ing and decoding are conducted on the multi- level vocabularies. Since VPN is able to lever- age rich encoding and decoding information with multi-level vocabularies, it has the po- tential to generate better responses. Experi- ments on English Twitter and Chinese Wei- bo datasets demonstrate that VPN remarkably outperforms strong baselines. 1 Introduction As one of the long-term goals in AI and NLP, au- tomatic conversation devotes to constructing auto- matic dialogue systems to communicate with hu- mans (Turing, 1950). Benefited from large-scale human-human conversation data available on the Internet, data-driven dialog systems have attracted increasing attention of both academia and industry (Ritter et al., 2011; Shang et al., 2015a; Vinyals and Le, 2015; Li et al., 2016a,c, 2017). Recently, a popular approach to build dialog engines is to learn a response generation mod- el within an encoder-decoder framework such as sequence-to-sequence (Seq2Seq) model (Cho et al., 2014a). In such a framework, an encoder transforms the source sequence into hidden vec- tors, and a decoder generates the targeted se- quence based on the encoded vectors and previ- s Decoders Concepts Meanings Words Distilled Concepts Coarse Meanings Grounding Words Distilled Concepts Coarse Meanings Grounding Words Multi-Pass Encoders Multi-Pass Decoders Raw Words High-Level Encoder Low-Level Encoder Raw-Word Encoder High-Level Decoder Low-Level Decoder Raw-Word Decoder Multi-Level Vocabularies High-Level Clusters Low-Level Clusters Figure 1: Vocabulary pyramid networks for response generation. The dialogue input (context) and output (response) are represented by multi-level vocabularies (e.g., raw words, low-level clusters and high-level clus- ters) and then processed by multi-pass encoder and de- coder. ously generated words. In this process, the en- coder and decoder share a vocabulary (word list)1, and the targeted words are typically performed by a softmax classifier over the vocabulary word-by- word. However, such typical Seq2Seq model is prone to generate safe and repeated responses, such as “Me too” and “I don’t know”. In addition to the exposure bias issue2, the main reasons of this …

Content truncated to 500 words