TR at SemEval-2020 Task 4: Exploring the Limits of Language-model-based Common Sense Validation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 601–608 Barcelona, Spain (Online), December 12, 2020. 601 TR at SemEval-2020 Task 4: Exploring the Limits of Language-model-based Common Sense Validation Don Teo Center for AI and Cognitive Computing Thomson Reuters Toronto, Canada don.teo@tr.com Abstract In this paper, we present our submission for subtask A of the Common Sense Validation and Explanation (ComVE) shared task. We examine the ability of large-scale pre-trained language models to distinguish commonsense from non-commonsense statements. We also explore the utility of external resources that aim to supplement the world knowledge inherent in such language models, including commonsense knowledge graph embedding models, word concreteness ratings, and text-to-image generation models. We find that such resources provide insignificant gains to the performance of fine-tuned language models. We also provide a qualitative analysis of the limitations of the language model fine-tuned to this task. 1 Introduction The task of assimilating general world knowledge from textual data for the purpose of commonsense reasoning and inference has been a long-standing challenge in natural language understanding (Davis, 1990; Schubert, 2002). A general approach to this problem that has gained recent popularity is the use of neural-network-based language models (LM) (Bengio et al., 2003). Such models, when trained on massive amounts of diverse text corpora, have been found to capture implicitly a remarkable amount of commonsense knowledge (Trinh and Le, 2018). Moreover, recent advances in training deep contextualized word representations using language-model-type objectives over large text corpora have substantially improved the state-of-the-art performance on a wide variety of natural language understanding tasks (Peters et al., 2018; Radford, 2018; Devlin et al., 2019; Yang et al., 2019; Raffel et al., 2019), including an even greater ability to capture commonsense knowledge (Zhou et al., 2019b; Porada et al., 2019). Despite this success, a significant gap remains between the performance of such pre-trained LMs and human-level performance on commonsense reasoning tasks. In fact, there is evidence suggesting that current models fail to learn factual knowledge effectively (Poerner et al., 2019) and have difficulty with a variety of basic reasoning abilities necessary for commonsense inference (Talmor et al., 2019a; Kassner and Sch¨utze, 2019). This motivates us to explore supplementary resources aimed at augmenting the world knowledge inherent in pre-trained LMs. In this paper, we explore the ability of a state-of-the-art pre-trained LM in tackling the ComVE subtask A (Wang et al., 2020a). In particular, given a pair of sentences, we evaluate the model’s ability in identifying which sentence least agrees with common sense. We consider the model from both a language modeling and a supervised learning perspective. Moreover, we explore the utility of additional resources meant to augment the perceptual world knowledge of language models in solving this task. Finally, we provide a categorization of the types of sentences that our best performing model struggles against. 2 Related Work A variety of datasets have been created to examine a system’s general commonsense knowledge and inference capability through such tasks as anaphora resolution (Levesque et al., 2012; Sakaguchi …

Content truncated to 500 words