COD3S: Diverse Generation with Discrete Semantic Signatures

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5199–5211, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5199 COD3S: Diverse Generation with Discrete Semantic Signatures Nathaniel Weir1 Jo˜ao Sedoc2 Benjamin Van Durme1 1Department of Computer Science, Johns Hopkins University 2Department of Technology, Operations, and Statistics, New York University {nweir, vandurme}@jhu.edu, jsedoc@stern.nyu.edu Abstract We present COD3S, a novel method for gen- erating semantically diverse sentences using neural sequence-to-sequence (seq2seq) mod- els. Conditioned on an input, seq2seq mod- els typically produce semantically and syntac- tically homogeneous sets of sentences and thus perform poorly on one-to-many sequence gen- eration tasks. Our two-stage approach im- proves output diversity by conditioning gen- eration on locality-sensitive hash (LSH)-based semantic sentence codes whose Hamming dis- tances highly correlate with human judgments of semantic textual similarity. Though it is gen- erally applicable, we apply COD3S to causal generation, the task of predicting a proposi- tion’s plausible causes or effects. We demon- strate through automatic and human evalua- tion that responses produced using our method exhibit improved diversity without degrading task performance. 1 Introduction Open-ended sequence generation problems such as dialog, story generation, image captioning, or causal generation pose a practical challenge to neu- ral sequence-to-sequence (seq2seq) models, as they necessitate a diverse set of predicted outputs. The typical sampling method for seq2seq decoding is beam search, which produces a set of candidate se- quences that generally have high syntactic, lexical, and semantic overlap. Recent methods for improved diversity genera- tion make slight modifications to the neural archi- tecture or beam search algorithm (Xu et al., 2018; Li et al., 2016b), or impose lexical constraints during decoding (Post and Vilar, 2018; Hu et al., 2019a). Shu et al. (2019) propose the use of sen- tence codes, a technique in which generation is conditioned on a discrete code that aims to induce diversity in syntax or semantics. While their ap- proach is effective for syntactic codes, it is less so for semantics. Figure 1: Overview of the COD3S method. In training (a), the target side is prefixed with a discrete signature computed using locality-sensitive hashing (LSH) of the target’s SBERT embedding. At inference (b), a beam search is conditioned on each of k decoded signatures. In this work, we introduce an improved method for diverse generation conditioned on inferred sen- tence codes that explicitly capture meaningful se- mantic differences. We use the contextual sen- tence embeddings from Sentence-BERT (SBERT; Reimers and Gurevych, 2019), the cosine distances between which correlate highly with human scalar judgments of semantic textual similarity (STS). We construct discrete codes from these embeddings us- ing locality-sensitive hashing (Indyk and Motwani, 1998; Charikar, 2002), producing short binary sig- natures whose Hamming distances well-preserves the cosine distances between inputs. Our method induces a bitwise hierarchy of se- mantic bins whose similarities in signature imply similarities in semantics. Conditioning generation on a signature as a target-side prefix indicates the bin into which the generated sequence falls. We implement a two-stage decoding process that (1) in- …

Content truncated to 500 words