A-TIP: Attribute-aware Text Infilling via Pre-trained Language Model

Proceedings of the 29th International Conference on Computational Linguistics, pages 5857–5869 October 12–17, 2022. 5857 A-TIP: Attribute-aware Text Infilling via Pre-trained Language Model Dongyuan Li, Jingyi You, Kotaro Funakoshi, Tokyo Institute of TechnologyManabu Okumura {lidy,youjy,funakoshi,oku}@lr.pi.titech.ac.jp Abstract Text infilling aims to restore incomplete texts by filling in blanks, which has attracted more at- tention recently because of its wide application in ancient text restoration and text rewriting. However, attribute-aware text infilling is yet to be explored, and existing methods seldom focus on the infilling length of each blank or the number/location of blanks. In this paper, we propose an Attribute-aware Text Infilling method via a Pre-trained language model (A- TIP), which contains a text infilling component and a plug-and-play discriminator. Specifically, we first design a unified text infilling compo- nent with modified attention mechanisms and intra- and inter-blank positional encoding to better perceive the number of blanks and the infilling length for each blank. Then, we pro- pose a plug-and-play discriminator to guide generation towards the direction of improving attribute relevance without decreasing text flu- ency. Finally, automatic and human evaluations on three open-source datasets indicate that A- TIP achieves state-of-the-art performance com- pared with all baselines. 1 Introduction Originating from Cloze tests (Taylor, 1953), text infilling aims to fill in missing blanks in a sentence or paragraph by making use of the preceding and subsequent texts. For example, given two infilling tasks E1 and E2 in Fig.1, text infilling models are supposed to provide fine-grained control over the location of any number of blanks and infill a vari- able number of missing tokens for each blank. Text infilling has been gaining increasing attention in a number of prevailing research fields, including ancient text restoration (Lazar et al., 2021), text editing and rewriting (Su et al., 2021), and conver- sation generation (Ou et al., 2021). However, current text infilling methods are based only on bidirectional semantic constraints (Ou et al., 2021), and other abundant attribute-based E1: Sentiment Infilling SST-5 Dataset Watching these [Mask] is both [Mask] and [Mask]. Roberta BLK Watching these kids is both funny and heartbreaking too. Watching these teams is both inspiring and the action. A-TIP with Positive Relevance Infilling. Attribute set: c = {Positive} Watching these performances is both inspiring and artfully mesmerizing. A-TIP with Negative Relevance Infilling. Attribute set: c = {Negative} Watching these shows is both boring and disgusting me much. E2: Expert Knowledge Infilling Abstract Dataset [Mask] of [Mask] and [Mask] of their [Mask]. TIGS Systems of and control and capability of their distance. BERT One of her friends and one of their friends. A-TIP with Computer Science Relevance Infilling. Attribute set: c = {CS} Analysis of data sources and functions of their programs. A-TIP with Math Relevance Infilling. Attribute set: c = {Math} Introduction of randomness matrices and decomposition of their method. Figure 1: A-TIP can generate more fluent, diverse and attribute relevant infilling content in two examples. constraints, e.g., sentiment and topics, remain to be studied. In reality, infilling attribute-aware con- tent …

Content truncated to 500 words