Reweighting Strategy Based on Synthetic Data Identification for Sentence Similarity

Proceedings of the 29th International Conference on Computational Linguistics, pages 4853–4863 October 12–17, 2022. 4853 Reweighting Strategy based on Synthetic Data Identification for Sentence Similarity Taehee Kim†∗, ChaeHun Park∗, Jimin Hong, Radhika Dua, Edward Choi and Jaegul Choo KAIST AI, Letsur† {taeheekim,ddehun,jimmyh,radhikadua,edwardchoi,jchoo}@kaist.ac.kr Abstract Semantically meaningful sentence embeddings are important for numerous tasks in natural lan- guage processing. To obtain such embeddings, recent studies explored the idea of utilizing synthetically generated data from pretrained language models (PLMs) as a training corpus. However, PLMs often generate sentences much different from the ones written by human. We hypothesize that treating all these synthetic ex- amples equally for training deep neural net- works can have an adverse effect on learning semantically meaningful embeddings. To ana- lyze this, we first train a classifier that identifies machine-written sentences, and observe that the linguistic features of the sentences identified as written by a machine are significantly dif- ferent from those of human-written sentences. Based on this, we propose a novel approach that first trains the classifier to measure the impor- tance of each sentence. The distilled informa- tion from the classifier is then used to train a re- liable sentence embedding model. Through ex- tensive evaluation on four real-world datasets, we demonstrate that our model trained on syn- thetic data generalizes well and outperforms the existing baselines.1 1 Introduction High-quality sentence embeddings are essential in diverse applications of natural language pro- cessing (Cer et al., 2018; Reimers and Gurevych, 2019), including semantic textual similarity (Cer et al., 2017) and paraphrase identification (Dolan and Brockett, 2005). Unfortunately, obtaining a large amount of human-annotated datasets to train a sentence embedding model is difficult and expen- sive. To address this, Schick and Schütze (2021) recently introduced a method, DINO, to train a ∗ Equal contribution 1Our implementation is publicly available at https://github.com/ddehun/coling2022_ reweighting_sts. sentence embedding model on synthetic data gen- erated from pretrained language models (PLMs). Despite the effectiveness and scalability of DINO, however, the difference between machine-written and human-written examples has not been carefully investigated. In other words, the study on the im- pact of treating all these synthetic examples equally during training remains under-explored. To this end, we first conduct an in-depth analysis to demonstrate the shift of synthetic samples from the human-written sentences. In particular, we train a classifier (i.e., Synthetic Data Identification (SDI) model) that identifies synthetic data from human- written sentences and observe that the linguistic fea- tures of the sentences predicted as machine-written are much different from the human-written sen- tences compared to the linguistic features of the sentences predicted as human-written. Based on this analysis, we propose a simple method, Reweighting Loss based on Importance of Machine-written SEntence (RISE), which first utilizes the trained SDI model to measure the im- portance of each sentence in learning semantically meaningful sentence embeddings for sentence sim- ilarity tasks. We then utilize this distilled informa- tion from the SDI model to reweight the loss of each synthetic example during training. We extensively evaluate our method on multiple …

Content truncated to 500 words