Towards Few-shot Fact-Checking via Perplexity

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1971–1981 June 6–11, 2021. ©2021 Association for Computational Linguistics 1971 Towards Few-Shot Fact-Checking via Perplexity Nayeon Lee†∗ Yejin Bang†∗ Andrea Madotto† Madian Khabsa§ Pascale Fung† †Hong Kong University of Science and Technology §Facebook AI {nyleeaa,yjbang}@connect.ust.hk Abstract Few-shot learning has drawn researchers’ at- tention to overcome the problem of data scarcity. Recently, large pre-trained language models have shown great performance in few- shot learning for various downstream tasks, such as question answering and machine trans- lation. Nevertheless, little exploration has been made to achieve few-shot learning for the fact-checking task. However, fact-checking is an important problem, especially when the amount of information online is growing ex- ponentially every day. In this paper, we pro- pose a new way of utilizing the powerful trans- fer learning ability of a language model via a perplexity score. The most notable strength of our methodology lies in its capability in few- shot learning. With only two training samples, our methodology can already outperform the Major Class baseline by more than an abso- lute 10% on the F1-Macro metric across multi- ple datasets. Through experiments, we empir- ically verify the plausibility of the rather sur- prising usage of the perplexity score in the con- text of fact-checking and highlight the strength of our few-shot methodology by comparing it to strong fine-tuning-based baseline mod- els. Moreover, we construct and publicly re- lease two new fact-checking datasets related to COVID-19. 1 Introduction Few-shot learning is being actively explored to overcome the heavy dependence on large-scale la- beled data that serves as a crucial bottleneck to machine learning models. Recently, researchers have explored few-shot learning that leverages the powerful transfer learning ability of pre-trained large language models (LMs) in various NLP tasks. Petroni et al. demonstrated that an LM serves as a good zero-shot learner on the question-answering task due to its encoded commonsense knowledge. ∗ Equal contribution. Figure 1: Illustration of our simple yet effec- tive perplexity-based approach. Few-shot data sam- ples are used to find the optimal perplexity thresh- old th that separates Unsupported claims from Supported claims. Going further, Brown et al. illustrated the impres- sive potential of LMs as strong zero-shot and few- shot learners across translation, commonsense rea- soning and natural language inference (NLI). How- ever, little or no exploration has been made on few- shot learning in the fact-checking domain, which is a timely and important task in which data-scarcity is particularly problematic. Previous works have proposed different ways of leveraging LMs to conduct zero- or few-shot learn- ing. One common approach is to query the LM for the missing token (i.e., “answer”) for the zero- shot question-answering task (Petroni et al., 2019; Brown et al., 2020) by transforming questions into a form of statement. Another approach is to adopt an in-context learning approach where the input context of the LM is carefully crafted to control the output. For example, a natural language …

Content truncated to 500 words