On the Importance of Delexicalization for Fact Verification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3413–3418, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3413 On the Importance of Delexicalization for Fact Verification Sandeep Suntwal∗, Mithun Paul∗, Rebecca Sharp, Mihai Surdeanu University of Arizona, Tucson, Arizona, USA {sandeepsuntwal,mithunpaul,bsharp,msurdeanu}@email.arizona.edu Abstract While neural networks produce state-of-the-art performance in many NLP tasks, they gener- ally learn from lexical information, which may transfer poorly between domains. Here, we in- vestigate the importance that a model assigns to various aspects of data while learning and mak- ing predictions, specifically, in a recognizing textual entailment (RTE) task. By inspecting the attention weights assigned by the model, we confirm that most of the weights are assigned to noun phrases. To mitigate this dependence on lexicalized information, we experiment with two strategies of masking. First, we replace named entities with their corresponding seman- tic tags along with a unique identifier to in- dicate lexical overlap between claim and evi- dence. Second, we similarly replace other word classes in the sentence (nouns, verbs, adjec- tives, and adverbs) with their super sense tags (Ciaramita and Johnson, 2003). Our results show that, while performance on the in-domain dataset remains on par with that of the model trained on fully lexicalized data, it improves considerably when tested out of domain. For example, the performance of a state-of-the-art RTE model trained on the masked Fake News Challenge (Pomerleau and Rao, 2017) data and evaluated on Fact Extraction and Verification (Thorne et al., 2018) data improved by over 10% in accuracy score compared to the fully lexicalized model. 1 Introduction Neural networks (NNs) play a key role in most modern natural language processing (NLP) sys- tems, obtaining state-of-the-art (SOA) perfor- mance (Devlin et al., 2018; Sun et al., 2018; Bohnet et al., 2018) in many complex tasks, e.g., recog- nizing textual entailment (Kim et al., 2018), fake news detection (Baird et al., 2017) and fact verifi- cation (Nie et al., 2018). ∗Equal Contribution However, these models depend heavily on lexi- cal information that may transfer poorly between different domains. For example, in early experi- ments in the fact verification space, we observed that out of all the 34 statements containing the phrase “American Author,”, 31 (91%) belonged to one class label. Such information could be mean- ingful say, in the literature domain, but transfers poorly to other domains such as science or enter- tainment. In this work we aim to: (a) understand and esti- mate the importance that a neural network assigns to various aspects of the data while learning and making predictions, and (b) learn how to control for unnecessary lexicalization. Here we focus on the recognizing textual entailment (RTE) task (Dagan et al., 2013), and its application to fact verification (Thorne et al., 2018; Pomerleau and Rao, 2017). RTE is the task of determining if one piece of text can be plausibly inferred from another. In the Fact Extraction and Verification …

Content truncated to 500 words