Top-Down Structurally-Constrained Neural Response Generation with Lexicalized Probabilistic Context-Free Grammar

Proceedings of NAACL-HLT 2019, pages 3762–3771 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3762 Top-Down Structurally-Constrained Neural Response Generation with Lexicalized Probabilistic Context-Free Grammar Wenchao Du Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213 wenchaod@cs.cmu.edu Alan W Black Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213 awb@cs.cmu.edu Abstract We consider neural language generation un- der a novel problem setting: generating the words of a sentence according to the order of their first appearance in its lexicalized PCFG parse tree, in a depth-first, left-to-right man- ner. Unlike previous tree-based language gen- eration methods, our approach is both (i) top- down and (ii) explicitly generating syntactic structure at the same time. In addition, our method combines neural model with symbolic approach: word choice at each step is con- strained by its predicted syntactic function. We applied our model to the task of dialog re- sponse generation, and found it significantly improves over sequence-to-sequence baseline, in terms of diversity and relevance. We also investigated the effect of lexicalization on lan- guage generation, and found that lexicalization schemes that give priority to content words have certain advantages over those focusing on dependency relations. 1 Introduction Neural encoder-decoder architectures have shown promise and become very popular for natural language generation. Over the past few years, there has seen a surging interest in sequence-to- sequence learning for dialog response generation using neural encoder-decoder models (Vinyals and Le, 2015; Serban et al., 2017). Typically, an encoder encodes conversational context (source side) information into vector representations, and a decoder auto-regressively generates word tokens conditioned on the source vectors and previously generated words. Two problems arise with the standard left-to- right decoding mechanism. First, no future in- formation is available at any step of the decoding process, while the study of linguistic dependency structure shows that certain words depend on the others that come right to them. Second, preced- ing words define the context for following words in left-to-right, auto-regressive language models, while linguistic theories may prefer other hierar- chies (e.g., adjectives modifying nouns, adverbs modifying verbs). Psycho-linguistics studies also suggest that human may first generate the abstract representation of the things to say, and then lin- earize them into sentences (Dell et al., 1999). Therefore, it is appealing to consider language generation in alternative orders. This poses a greater challenge because a mechanism in ex- tra to word generation is needed for deciding the position of each word. Some recent works adopt a syntax-free approach to address this prob- lem. (Mehri and Sigal, 2018) proposed a middle- out decoder that starts from the middle of sen- tences and finishes the rest in forward and back- ward directions. (Mou et al., 2016) and (Li and Sun, 2018) start with one or two predicted key- words and generate the rest of sentences in a sim- ilar fashion. Others incorporate tree structures without syntactic relations and categories. (Zhou et al., 2018) canonicalizes the dependency struc- tures of sentences into ternary trees, and gener- …

Content truncated to 500 words