Coordination Generation via Synchronized Text-Infilling

Proceedings of the 29th International Conference on Computational Linguistics, pages 5914–5924 October 12–17, 2022. 5914 Coordination Generation via Synchronized Text-Infilling Hiroki Teranishi and Yuji Matsumoto RIKEN Center for Advanced Intelligence Project {hiroki.teranishi, yuji.matsumoto}@riken.jp Abstract Generating synthetic data for supervised learn- ing from large-scale pre-trained language mod- els has enhanced performances across several NLP tasks, especially in low-resource scenar- ios. In particular, many studies of data aug- mentation employ masked language models to replace words with other words in a sentence. However, most of them are evaluated on sen- tence classification tasks and cannot immedi- ately be applied to tasks related to the sentence structure. In this paper, we propose a simple yet effective approach to generating sentences with a coordinate structure in which the boundaries of its conjuncts are explicitly specified. For a given span in a sentence, our method embeds a mask with a coordinating conjunction in two ways (“X and <mask>”, “<mask> and X”) and forces masked language models to fill the two blanks with an identical text. To achieve this, we introduce decoding methods for BERT and T5 models with the constraint that predic- tions for different masks are synchronized. Fur- thermore, we develop a training framework that effectively selects synthetic examples for the su- pervised coordination disambiguation task. We demonstrate that our method produces promis- ing coordination instances that provide gains for the task in low-resource settings. 1 Introduction Pre-trained language models (LMs) have brought a large shift from preparing a substantial number of training examples and learning task-specific rep- resentations to collecting tons of unlabeled text and learning task-agnostic representations. For instance, fine-tuning LMs, such as BERT (De- vlin et al., 2019) and BART (Lewis et al., 2020), has improved performances of many natural lan- guage processing (NLP) tasks, even when us- ing a small amount of training data. This paradigm shift has led to many challenges to ex- ploit knowledge of LMs rather than manually an- notated data, including zero- or few-shot learning (Radford et al., 2019; Brown et al., 2020; Shin et al., 2020; Gao et al., 2021) and data augmenta- tion (Wu et al., 2019; Gao et al., 2019; Anaby-Tavor et al., 2020; Kumar et al., 2020). However, many of these studies focus on sentence classification tasks, such as question answering and sentiment analysis, and thus are not applicable to structured prediction tasks, such as syntactic parsing. Although some studies propose data augmentation methods that manipulate the structure of a sentence (¸Sahin and Steedman, 2018; Shi et al., 2021), producing syn- tactic structures using recent pre-trained LMs has not been explored. As an attempt to produce syntactic structures using LMs, we explore approaches to generating coordinate structures, in which two or more ele- ments, known as conjuncts, are linked by a coor- dinating conjunction (coordinator), and yielding the corresponding annotation that explicitly marks which elements in the sentence are coordinated. Co- ordination frequently appears in natural language sentences and causes ambiguities, leading to er- rors in many NLP tasks (Hara et al., …

Content truncated to 500 words