Mixture Content Selection for Diverse Sequence Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3121–3131, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3121 Mixture Content Selection for Diverse Sequence Generation Jaemin Cho1∗ Minjoon Seo2,3 Hannaneh Hajishirzi1,3 Allen Institute for AI1 Clova AI, NAVER2 University of Washington3 heythisischo@gmail.com {minjoon,hannaneh}@cs.washington.edu Abstract Generating diverse sequences is important in many NLP applications such as question generation or summarization that exhibit se- mantically one-to-many relationships between source and the target sequences. We present a method to explicitly separate diversification from generation using a general plug-and-play module (called SELECTOR) that wraps around and guides an existing encoder-decoder model. The diversification stage uses a mixture of experts to sample different binary masks on the source sequence for diverse content se- lection. The generation stage uses a stan- dard encoder-decoder model given each se- lected content from the source sequence. Due to the non-differentiable nature of discrete sampling and the lack of ground truth la- bels for binary mask, we leverage a proxy for ground-truth mask and adopt stochastic hard-EM for training. In question genera- tion (SQuAD) and abstractive summarization (CNN-DM), our method demonstrates signifi- cant improvements in accuracy, diversity and training efficiency, including state-of-the-art top-1 accuracy in both datasets, 6% gain in top-5 accuracy, and 3.7 times faster training over a state-of-the-art model. Our code is pub- licly available at https://github.com/ clovaai/FocusSeq2Seq. 1 Introduction Generating target sequences given a source se- quence has applications in a wide range of prob- lems in NLP with different types of relationships between the source and target sequences. For in- stance, paraphrasing or machine translation ex- hibit a one-to-one relationship because the source and the target should carry the same meaning. On the other hand, summarization or question gen- eration exhibit one-to-many relationships because ∗Most work done during internship at Clova AI. Source Passage: in december 1878 , tesla left graz and severed all relations with his family to hide the fact that he dropped out of school . Target: what did tesla do in december 1878? Focus 1: in december 1878 , tesla left graz and severed all relations with his family to hide the fact that he dropped out of school. (Ours) ⇒ what did tesla do? Focus 2: in december 1878 , tesla left graz and severed all relations with his family to hide the fact that he dropped out of school. (Ours) ⇒ what did tesla do in december 1878? Focus 3: in december 1878 , tesla left graz and severed all relations with his family to hide the fact that he dropped out of school . (Ours) ⇒ what did tesla do to hide he dropped out of school? Figure 1: Sample questions produced by our method from given passage-answer pair (answer is underlined). Our method generates diverse questions, by selecting different tokens to focus (colored) in contrast to 3- mixture decoder (Shen et al., 2019) that generates 3 identical questions: “what did …

Content truncated to 500 words