ADEPT: An Adjective-Dependent Plausibility Task

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7117–7128 August 1–6, 2021. ©2021 Association for Computational Linguistics 7117 ADEPT: An Adjective-Dependent Plausibility Task Ali Emami1, Ian Porada1, Alexandra Olteanu2, Kaheer Suleman2, Adam Trischler2, and Jackie Chi Kit Cheung1 1Mila, McGill University 2Microsoft Research Montr´eal {ali.emami, ian.porada}@mail.mcgill.ca {alexandra.olteanu, adam.trischler, kasulema}@microsoft.com jcheung@cs.mcgill.ca Abstract A false contract is more likely to be re- jected than a contract is, yet a false key is less likely than a key to open doors. While correctly interpreting and assessing the ef- fects of such adjective-noun pairs (e.g., false key) on the plausibility of given events (e.g., opening doors) underpins many natural lan- guage understanding tasks, doing so often re- quires a significant degree of world knowl- edge and common-sense reasoning. We intro- duce ADEPT – a large-scale semantic plausi- bility task consisting of over 16 thousand sen- tences that are paired with slightly modified versions obtained by adding an adjective to a noun. Overall, we find that while the task ap- pears easier for human judges (85% accuracy), it proves more difficult for transformer-based models like RoBERTa (71% accuracy). Our experiments also show that neither the adjec- tive itself nor its taxonomic class suffice in de- termining the correct plausibility judgement, emphasizing the importance of endowing auto- matic natural language understanding systems with more context sensitivity and common- sense reasoning. 1 Introduction Discerning the varying effects of adjectival mod- ifiers on the reading of a sentence is critical in a variety of tasks involving natural language under- standing. Consider the following examples: (1) a. A [dead] monkey turns on a light switch. b. A [dead] leg has one foot. c. A [dead] leaf falls from a tree in autumn. The reading of these sentences with and without the modifier dead is notably different. The plausi- bility judgement of the event where a monkey turns on a light switch decreases when the adjectival modifier dead is added, while in the 1b or 1c exam- ples, adding the same modifier leads to no change or an increase in event plausibility, respectively. This observation has important ramifications for many NLP applications like information extrac- tion (IE) and recognizing textual entailment (RTE), where solutions have often relied on normative rules that group the effects of adjectives according to either the adjective or its taxonomic class (Mc- Nally and Boleda, 2004; Amoia and Gardent, 2007; McCrae et al., 2014). These taxonomies distinguish adjectives like false, dead, alleged (non-subsective) from others like red, large, or valid (subsective). Specifically, while the 1a example may influ- ence systems to adopt the rule that adding a non- subsective adjective like dead to a noun leads to a decrease in plausibility, the other examples suggest a conflicting rule. Distinguishing the effects of dif- ferent adjectives (beyond just their denotation) may thus require common-sense and world knowledge. Powerful, massively pre-trained language mod- els (LMs) have pushed the performance on vari- ous natural …

Content truncated to 500 words