Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2061–2073 August 1–6, 2021. ©2021 Association for Computational Linguistics 2061 Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions Nam Do Brown University Providence, RI, USA nam do@alumni.brown.edu Ellie Pavlick Brown University Providence, RI, USA ellie pavlick@brown.edu Abstract Previous studies have argued that pre-trained language models encode commonsense rela- tional knowledge (e.g. that apples are ed- ible). However, simultaneous work has re- vealed that such models are often insensitive to context, even ignoring overt contextual cues such as negations. In this paper, we investigate whether masked language models (the BERT family) can move beyond naive associative bi- ases (e.g., apple → edible) when the context warrants (e.g. ranking inedible higher when presented with the information that the apple is rotten). We introduce the WINOVENTI pro- cedure, which adversarially exploits generic associations in masked language models to cre- ate model-specific Winograd-style entailment schemas. Using our constructed WINOVENTI challenges set of over 2, 000 schemas, we show that language models in the BERT fam- ily experience a steep drop in performance on prompts that require them to pick answers which require reasoning about context (e.g., from 89.8% to 18.4% for BERTLARGE). We present evidence that language models exhibit different associative biases, suggesting a need for future work in developing and analyzing frameworks similar to WINOVENTI that are tuned to model-specific weaknesses. 1 Introduction Humans exhibit commonsense knowledge through their ability to identify generics (e.g., that a dog has four legs) while still recognizing that exceptions to such rules are possible (e.g., that there are cases of three-legged dogs) (Greenberg, 2007), and that the probability of such exceptions can vary based on the context (e.g. “the dog is running” vs. “the dog is hobbling”). A prerequisite to comparing a machine’s performance to human intelligence is, hence, the verification that machines can exhibit a sensitivity to context that would allow them to perform as well on cases that require reasoning about exceptions as on cases that require recalling generic associations. Recent work (Petroni et al., 2019) has shown that large pretrained language models, in partic- ular Masked Language Models (MLMs) such as BERT (Devlin et al., 2018) are competent at asso- ciating entities with their common characteristics. For example, BERTLARGE readily recalls apple → edible and charcoal → hot. However, as demon- strated by Ettinger (2020) and Kassner and Sch¨utze (2019), BERT is insensitive to various overt con- textual cues, notably negation. For example, given the context “The shower is ,” BERTLARGE pre- dicts the words “cold”, “long”, and “empty”, the same top 3 predictions it makes given the context “The shower is not ”. Such results suggest that while language models like BERT capture many commonsense patterns, such success might be lim- ited to inferences involving common generaliza- tions (appearing in an affirmative context, or using common lexical associations) and not those involv- ing exceptions (appearing in a negative context, or requiring the models to choose less frequently associated lexical items). In …

Content truncated to 500 words