Data and Model Distillation as a Solution for Domain-transferable Fact Verification

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4546–4552 June 6–11, 2021. ©2021 Association for Computational Linguistics 4546 Data and Model Distillation as a Solution for Domain-transferable Fact Verification Mitch Paul Mithun, Sandeep Suntwal, Mihai Surdeanu University of Arizona, Tucson, Arizona, USA {mithunpaul,sandeepsuntwal,msurdeanu}@email.arizona.edu Abstract While neural networks produce state-of-the- art performance in several NLP tasks, they gen- erally depend heavily on lexicalized informa- tion, which transfer poorly between domains. We present a combination of two strategies to mitigate this dependence on lexicalized infor- mation in fact verification tasks. We present a data distillation technique for delexicaliza- tion, which we then combine with a model distillation method to prevent aggressive data distillation. We show that by using our solu- tion, not only does the performance of an ex- isting state-of-the-art model remain at par with that of the model trained on a fully lexicalized data, but it also performs better than it when tested out of domain. We show that the tech- nique we present encourages models to extract transferable facts from a given fact verification dataset. 1 Introduction Neural networks have matched, and in several cases even surpassed, human performance in several su- pervised learning problems. However, such suc- cesses come at a cost. These neural networks typ- ically need a great deal of human support in the form of man power required for curating domain specific datasets. Further, it has been shown (Gu- rurangan et al., 2018; Poliak et al., 2018; Thorne and Vlachos, 2020) that several such models de- pend heavily on certain statistical nuances found in these datasets, information that transfers poorly be- tween domains. The ideal solution to this problem is the creation of models that do not rely on such statistical nuances in the given datasets, but instead encode the true underlying semantics of the task, that are in turn transferable to other domains. Fact verification is the task of verifying the truth- fulness of claims by estimating their assertions against credible evidences. Specifically, given a pair of claim and evidence statements, they have to be classified into one of the 3 class labels, agree, disagree, or neutral. Fact verification datasets, which often constitute real life news articles, have the added advantage of being used in practical prob- lems such as fake news detection. More recently, several neural network models (Nie et al., 2020; Liu et al., 2020, inter alia) built on top of the transform- ers (Vaswani et al., 2017), have achieved excellent performance in fact verification tasks. However these methods are not devoid of the shortcomings that besiege other neural networks in natural language processing tasks. It has been shown that these approaches depend heavily on lex- ical artifacts that transfer poorly between domains (Panenghat et al., 2020; Karimi Mahabadi et al., 2020; Schuster et al., 2019). For example, Suntwal et al. (2019) observed that out of all the statements containing the phrase ‘American Author’ in the FEVER dataset (Thorne et …

Content truncated to 500 words