Modeling Noise in Paraphrase Detection

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4324–4332 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4324 Modeling Noise in Paraphrase Detection Teemu Vahtola, Eetu Sj¨oblom, J¨org Tiedemann, Mathias Creutz Department of Digital Humanities Faculty of Arts University of Helsinki Finland {teemu.vahtola, eetu.sjoblom, jorg.tiedemann, mathias.creutz}@helsinki.fi Abstract Noisy labels in training data present a challenging issue in classification tasks, misleading a model towards incorrect decisions during training. In this paper, we propose the use of a linear noise model to augment pre-trained language models to account for label noise in fine-tuning. We test our approach in a paraphrase detection task with various levels of noise and five different languages. Our experiments demonstrate the effectiveness of the additional noise model in making the training procedures more robust and stable. Furthermore, we show that this model can be applied without further knowledge about annotation confi- dence and reliability of individual training examples and we analyse our results in light of data selection and sampling strategies. Keywords: paraphrase detection, label noise 1. Introduction Large pre-trained language models have obtained great results in many NLP tasks in recent years by utilising the dominant pre-training and fine-tuning paradigm. The language models’ success on a specific down- stream task is to a great extent dependent on the quality and quantity of task-specific training data. A consider- able amount of annotated, high-quality training data is often necessary even for fine-tuning a large pre-trained language model in order for it to perform well on a cer- tain downstream task. Consequently, different meth- ods are utilised to obtain sufficient amounts of training data in a cost-effective way leading to different levels of noise. In this paper, we study the use of an explicit noise model to address label noise – incorrectly classified training samples that mislead the model during train- ing. The noise model can be added to a neural ar- chitecture to make the training procedures more robust and predictable when noise enters the model. We ap- ply the task of paraphrase detection as an example to demonstrate the use of an additional noise layer that is attached to a language model during fine-tuning. Our contributions are: (1) An implementation of a noise model that extends pre-trained language mod- els to increase robustness when fine-tuning paraphrase detection with noisy training data. (2) A study that shows the effect of noise modeling with various de- grees of noise across five different languages. (3) Ex- periments that demonstrate the effectiveness of noise models without explicit information about annotation quality and label confidence. 2. Training with Label Noise Online platforms are often utilized for the acquisition of large quantities of annotations from non-experts. The quality of such crowdsourced human-made anno- tations has recently been questioned, for example, in machine translation evaluation (see, e.g., Graham et al. (2017); Toral et al. (2018); L¨aubli et al. (2020)). Annotators seem to assign higher scores to transla- tions that more accurately conform the sentence struc- …

Content truncated to 500 words