INSET: Sentence Infilling with INter-SEntential Transformer

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2502–2515 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2502 INSET: Sentence Infilling with INter-SEntential Transformer Yichen Huang (黄 黄 黄溢 溢 溢辰 辰 辰)12∗, Yizhe Zhang1∗, Oussama Elachqar1, Yu Cheng1 1Microsoft Corporation, Redmond, Washington 98052, USA 2Center for Theoretical Physics, MIT, Cambridge, Massachusetts 02139, USA yichuang@mit.edu, {yizzhang, ouelachq, yu.cheng}@microsoft.com Abstract Missing sentence generation (or sentence in- filling) fosters a wide range of applications in natural language generation, such as document auto-completion and meeting note expansion. This task asks the model to generate interme- diate missing sentences that can syntactically and semantically bridge the surrounding con- text. Solving the sentence infilling task re- quires techniques in natural language process- ing ranging from understanding to discourse- level planning to generation. In this paper, we propose a framework to decouple the chal- lenge and address these three aspects respec- tively, leveraging the power of existing large- scale pre-trained models such as BERT and GPT-2. We empirically demonstrate the effec- tiveness of our model in learning a sentence representation for generation and further gen- erating a missing sentence that fits the context. 1 Introduction Generating a span of missing tokens in a text chunk, known as “text infilling,” has attracted many atten- tions recently (Zhu et al., 2019; Song et al., 2019; Liu et al., 2019; Ippolito et al., 2019; Joshi et al., 2020). Here we study the related but somewhat dif- ferent task of “sentence infilling.” Specifically, as illustrated in Figure 1, intermediate sentences (or chunks of text) are removed from long-form text (e.g., paragraphs, documents), and the task is to generate the missing pieces that can smoothly blend into and fit the context both syntactically and se- mantically. The generation can be either based only on context, or based on both context and side in- formation such as constraint keywords. Compared with text infilling, sentence infilling requires the model to handle inter-sentential correlation and to reason about missing semantic information. Devel- oping models for sentence infilling can potentially ∗These authors contributed equally to this work. She was extremely happy with our hotel and we had a complimentary buffet. ... The food was just phenomenal! I can’t recall what everything was called, but we rolled out of there stuffed and happy. My husband had the rib eye dumpling as an appetizer and he said it was the best dumpling he has ever had. Beautiful beachside boutique hotel with great views and modern decoration. My favorite part about this hotel is definitely the restaurant, UVA. I recently visited UVA to attend a friend’s birthday party. ... Figure 1: Sentence infilling: generating an intermedi- ate sentence that provides a smooth semantic transition from the preceding to the following context. This ex- ample is generated by our model on the TripAdvisor dataset. The colors mark the correspondence between the generated sentence and the context. facilitate many text generation applications. Possi- ble scenarios include, but are not limited to: docu- ment …

Content truncated to 500 words