InFillmore: Frame-Guided Language Generation with Bidirectional Context

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 129–142 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 129 InFillmore: Frame-Guided Language Generation with Bidirectional Context Jiefu Ou†‡ Nathaniel Weir† Anton Belyy† Felix Yu Benjamin Van Durme Johns Hopkins University jouaa@connect.ust.hk {nweir,abel,fyu17,vandurme}@jhu.edu Abstract We propose a structured extension to bidirectional-context conditional language generation, or “infilling,” inspired by Frame Semantic theory (Fillmore, 1976). Guidance is provided through two approaches: (1) model fine-tuning, conditioning directly on observed symbolic frames, and (2) a novel extension to disjunctive lexically constrained decoding that leverages frame semantic lexical units. Au- tomatic and human evaluations confirm that frame-guided generation allows for explicit manipulation of intended infill semantics, with minimal loss in distinguishability from human-generated text. Our methods flexibly apply to a variety of use scenarios, and we provide an interactive web demo.1 1 Introduction A popular strategy for automatic story generation is to proceed in a coarse-to-fine manner: first by proposing a story plan, and then realizing it into natural language form using large pretrained neu- ral language models (Fan et al., 2018; Goldfarb- Tarrant et al., 2019). In this work, we study the use of FrameNet frames (Baker et al., 1998) as representational units for such plan guidance. In Frame Semantics (Fillmore, 1976; Fillmore and Baker, 2010), words evoke structural situation types (frames) that describe the common schematic relationships between lexical items. We hypothe- size that these structured types can be used to effec- tively induce the semantic content of text generated by increasingly powerful pretrained language mod- els, yielding a flexible, controllable and domain- general model for surface realization of story plans with a variety of dimensions for user guidance. †Corresponding authors. ‡ Work done during an internship at the Center for Lan- guage and Speech Processing, JHU. 1Codebase and demo available from https://nlp. jhu.edu/demos/infillmore. Figure 1: The proposed generation model, applied to the interactive story generation task. Similar to the ex- isting infilling models, a user can insert or rewrite text spans at any position in a story. With the proposed extension, generation can be guided via explicit frame semantic constraints, either provided manually or sug- gested by the model based on surrounding context. Based on this supposition, we fine-tune a recent infilling model (Donahue et al., 2020) with a frame- guided denoising objective. We contrast this ap- proach with a novel method for frame-guided gener- ation that modifies only the decoding step of a stan- dard language model through lexical manipulation. The idea originates from the annotation scheme of FrameNet, where each semantic frame is annotated with a set of evocative lexical units (LUs). We posit that it is possible to guide the model’s generation with frames without modifying its training proce- dure by instead lexically constraining its generation output to contain frame-associated LUs. Therefore, we develop an extension to lexically-constrained decoding that leverages LUs as ordered disjunc- tive constraint sets. Given a possibly multi-frame sequence and a generative model, our method en- 130 forces the generation of …

Content truncated to 500 words