Enconter: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3590–3599 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3590 ENCONTER: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer Lee-Hsun Hsieh jesus255221@gmail.com Yang-Yin Lee Singapore Management University Singapore yylee@smu.edu.sg Ee-Peng Lim eplim@smu.edu.sg Abstract Pretrained using large amount of data, autore- gressive language models are able to gener- ate high quality sequences. However, these models do not perform well under hard lex- ical constraints as they lack fine control of content generation process. Progressive insertion-based transformers can overcome the above limitation and efficiently generate a sequence in parallel given some input to- kens as constraint. These transformers how- ever may fail to support hard lexical con- straints as their generation process is more likely to terminate prematurely. The pa- per analyses such early termination problems and proposes the ENtity-CONstrained insertion TransformER (ENCONTER), a new insertion transformer that addresses the above pitfall without compromising much generation effi- ciency. We introduce a new training strat- egy that considers predefined hard lexical con- straints (e.g., entities to be included in the gen- erated sequence). Our experiments show that ENCONTER outperforms other baseline mod- els in several performance metrics rendering it more suitable in practical applications. 1 1 Introduction The field of Natural Language Generation (NLG) (Gatt and Krahmer, 2018) has seen signifi- cant improvements in recent years across many ap- plications such as neural machine translation (Bah- danau et al., 2015), text summarization (Chopra et al., 2016), poem generation (Zugarini et al., 2019) and recipe generation (H. Lee et al., 2020). Constrained text generation (CTG) is one of the challenging problems in NLG that is important to many real world applications but has not been well addressed. CTG imposes input constraints which may be in the form of objects expected to exist in 1Our code is available at https://github.com/ LARC-CMU-SMU/Enconter the generated text or rules over objects in the gener- ated text (Hokamp and Liu, 2017). The objects here can be entities, phrases, predefined nouns, verbs, or sentence fragments. The constraints can be cate- gorized into two types: (1) Hard-constraints which require mandatory inclusion of certain objects and complete compliance of given rules (Post and Vilar, 2018; Hu et al., 2019; Miao et al., 2019; Welleck et al., 2019; Zhang et al., 2020); and (2) Soft- constraints which allow the some constraint objects or rules to be not strictly enforced in the generated text (Qin et al., 2019; Tang et al., 2019). As autore- gressive models generate tokens from left to right, they cannot easily support constraints involving multiple input objects, hard-constrained text gen- eration therefore often requires non-autoregressive models. Recently, Zhang et al. (2020) proposed a non-autoregressive hard-constrained text genera- tion model (POINTER) that generates a text se- quence in a progressive manner using an insertion- transformer (Stern et al., 2019). To train an in- sertion transformer to generate a missing token between every two tokens in an input sequence, the training data is prepared …

Content truncated to 500 words