TIGS: An Inference Algorithm for Text Infilling with Gradient Search

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4146–4156 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4146 TIGS: An Inference Algorithm for Text Infilling with Gradient Search Dayiheng Liu†, Jie Fu‡, Pengfei Liu§, Jiancheng Lv†∗ †College of Computer Science, Sichuan University ‡Mila, IVADO, Polytechnique Montreal §School of Computer Science, Fudan University losinuris@gmail.com lvjiancheng@scu.edu.cn Abstract Text infilling is defined as a task for filling in the missing part of a sentence or para- graph, which is suitable for many real-world natural language generation scenarios. How- ever, given a well-trained sequential genera- tive model, generating missing symbols condi- tioned on the context is challenging for exist- ing greedy approximate inference algorithms. In this paper, we propose an iterative inference algorithm based on gradient search, which is the first inference algorithm that can be broadly applied to any neural sequence gener- ative models for text infilling tasks. We com- pare the proposed method with strong base- lines on three text infilling tasks with various mask ratios and different mask strategies. The results show that our proposed method is ef- fective and efficient for fill-in-the-blank tasks, consistently outperforming all baselines.1 1 Introduction Text infilling aims at filling in the missing part of a sentence or paragraph by making use of the past and future information around the miss- ing part, which can be used in many real-world natural language generation scenarios, for exam- ple, fill-in-the-blank image captioning (Sun et al., 2017), lexically constrained sentence generation (Liu et al., 2018b), missing value reconstruc- tion (e.g. for damaged or historical documents) (Berglund et al., 2015), acrostic poetry generation (Liu et al., 2018a), and text representation learning (Devlin et al., 2018). Text infilling is an under-explored challeng- ing task in the field of text generation. Re- cently, sequence generative models like sequence- to-sequence (seq2seq) models (Sutskever et al., ∗ Correspondence to Jiancheng Lv. 1Our code and data are available at https://github.com/dayihengliu/ Text-Infilling-Gradient-Search Input:'''Hey,'how'about'going'for'a'few'beers'after'dinner'?' Ground'Truth You'know'that'is tempting but is really not good for'our'fitness'. Seq2seq'+'Left-to-Right'Beam'Search You'know'that'I like it very much , let’s for'our'fitness'. Seq2seq'(backward)' +'Right-to-Left'Beam'Search You'know'that not going , it is really bad for'our'fitness'. Figure 1: Our key observation on text infilling for Di- alogue task. The inability of unidirectional BS to con- sider both the future and past contexts leads models to fill the blank with words that clash abruptly with the context around the blanks (see red circles). 2014; Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017) are widely used in text gen- eration tasks, including neural machine translation (Wu et al., 2016; Vaswani et al., 2017), image cap- tioning (Anderson et al., 2017), abstractive sum- marization (See et al., 2017), and dialogue gen- eration (Mei et al., 2017). Unfortunately, given a well-trained2 neural seq2seq model or uncon- ditional neural language model (Mikolov et al., 2010), it is a daunting task to directly apply it to text infilling task. As shown in Figure 1, we ob- serve that the infilled words should …

Content truncated to 500 words