Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks

Findings of the Association for Computational Linguistics: ACL 2022, pages 1887 - 1895 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks Ashutosh Kumar Indian Institute of Science, India ashutosh@iisc.ac.in Aditya Joshi SEEK, Australia aditya.m.joshi@gmail.com, ajoshi@seek.com.au Abstract While fine-tuning pre-trained models for downstream classification is the conventional paradigm in NLP, often task-specific nuances may not get captured in the resultant models. Specifically, for tasks that take two inputs and require the output to be invariant of the or- der of the inputs, inconsistency is often ob- served in the predicted labels or confidence scores. We highlight this model shortcoming and apply a consistency loss function to al- leviate inconsistency in symmetric classifica- tion. Our results show an improved consis- tency in predictions for three paraphrase detec- tion datasets without a significant drop in the accuracy scores. We examine the classification performance of six datasets (both symmetric and non-symmetric) to showcase the strengths and limitations of our approach. 1 Introduction Symmetric classification tasks involve two inputs and require that the model output should be inde- pendent of the order in which the two input texts are given. In other words, the output of the classi- fier should be the same and the confidence score must not be significantly different, if the inputs X and Y are instead supplied as Y and X. Paraphrase detection, multi-lingual semantic similarity are ex- amples of symmetric classification tasks. Although attention-based (Bahdanau et al., 2015; Vaswani et al., 2017) pre-trained language models have led to significant performance gains in multiple text classification tasks, they demonstrate a peculiar er- ratic behaviour on symmetric classification: incon- sistency. An example1 of inconsistency for para- phrase detection is shown in Figure 1. Additional examples can be found in the Appendix (Table 4). To alleviate such an inconsistency for symmetric classification tasks, we propose a simple additional 1Note that, while this particular example is based on our fine-tuned model, it will change depending on the trained model. The overall argument is valid, nonetheless. A provisional government or a revolutionary government has been declared several times by insurgent groups in the Philippines . A revolutionary government or a provisional government has been declared several times in the Philippines by insurgent groups . X Y X Y X Y Model Input Sequence (98.6) (88.3) (92.2) (87.9) Figure 1: Impact of reordering an example input pair (X and Y ) on standard fine-tuned BERT and BERT- with-consistency-loss . The pair are true paraphrases. and denote that the model predicted them to be paraphrases and not-paraphrases, respectively. Confi- dence scores are reported in brackets. Details in Sec- tion 1. drop-in fine-tuning objective, based on either the Kullback-Leibler (KL) or Jensen-Shannon (JS) di- vergence (or any f-divergence (Rubenstein et al., 2019)), to the cross-entropy loss for symmetric tasks. We refer to this as the consistency loss. The main contributions of this paper are: (a) Highlight inconsistency issues in symmetric classification tasks, (b) Describe a consistency …

Content truncated to 500 words