Plug-and-Play Conversational Models

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 212–222 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 212 Semantic Matching for Sequence-to-Sequence Learning Ruiyi Zhang3, Changyou Chen†, Xinyuan Zhang‡, Ke Bai3, Lawrence Carin3 3Duke University, †State University of New York at Buffalo, ‡ASAPP Inc. ryzhang.cs@gmail.com Abstract In sequence-to-sequence models, classical op- timal transport (OT) can be applied to seman- tically match generated sentences with target sentences. However, in non-parallel settings, target sentences are usually unavailable. To tackle this issue without losing the benefits of classical OT, we present a semantic matching scheme based on the Optimal Partial Trans- port (OPT). Specifically, our approach par- tially matches semantically meaningful words between source and partial target sequences. To overcome the difficulty of detecting active regions in OPT (corresponding to the words needed to be matched), we further exploit prior knowledge to perform partial matching. Ex- tensive experiments are conducted to evaluate the proposed approach, showing consistent im- provements over sequence-to-sequence tasks. 1 Introduction Sequence-to-sequence (Seq2Seq) models are widely used in various natural-language-processing tasks, such as machine translation (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015), text summarization (Rush et al., 2015; Chopra et al., 2016) and image captioning (Vinyals et al., 2015; Xu et al., 2015). Typically, these models are based on an encoder-decoder architecture, with an en- coder mapping a source sequence into a latent vec- tor, and a decoder translating the latent vector into a target sequence. The goal of a Seq2Seq model is to optimize this encoder-decoder network to gen- erate sequences close to the target. Therefore, a proper measure of the distance between sequences is crucial for model training. Wasserstein distance between two text se- quences, i.e., word-mover distance (Kusner et al., 2015), can serve as an effective regularizer for se- mantic matching in Seq2Seq models (Chen et al., 2019). Classical optimal transport models require that each piece of mass in the source distribution is transported to an equal-weight piece of mass in the target distribution. However, this requirement is too restrictive for Seq2Seq models, making direct applications inappropriate due to the following: (i) texts often have different lengths, and not every element in the source text corresponds an element in the target text. A good example is style transfer, where some words in the source text do not have corresponding words in the target text. (ii) it is reasonable to semantically match important words while neglecting some other words, e.g., conjunc- tion. In typical unsupervised models, text data are usually non-parallel in the sense that pairwise data are typically unavailable (Sutskever et al., 2014). Thus, both pairwise information inference and text generation must be performed in the same model with only non-parallel data. Classical OT is not applicable without target text sequences. However, partial target information is available, for exam- ple, the detected objects in an image should be de- scribed in its caption, and the content words when changing the style should be preserved. OT will fail in …

Content truncated to 500 words