Testing Large Language Models on Compositionality and Inference with Phrase-Level Adjective-Noun Entailment

Proceedings of the 29th International Conference on Computational Linguistics, pages 4084–4100 October 12–17, 2022. 4084 Testing Large Language Models on Compositionality and Inference with Phrase-Level Adjective-Noun Entailment Lorenzo Bertolini, Julie Weeds and David Weir University of Sussex Brighton, UK {l.bertolini, juliwe, d.j.weir}@sussex.ac.uk Abstract Previous work has demonstrated that pre- trained large language models (LLM) acquire knowledge during pre-training which enables reasoning over relationships between words (e.g, hyponymy) and more complex inferences over larger units of meaning such as sentences. Here, we investigate whether lexical entailment (LE, i.e. hyponymy or the is a relation between words) can be generalised in a compositional manner. Accordingly, we introduce PLANE (Phrase-Level Adjective-Noun Entailment), a new benchmark to test models on fine-grained compositional entailment using adjective-noun phrases. Our experiments show that knowledge extracted via In–Context and transfer learning is not enough to solve PLANE. However, a LLM trained on PLANE can generalise well to out–of–distribution sets, since the required knowledge can be stored in the representations of subwords (SW) tokens. 1 Introduction Composition and entailment are crucial features of human language and reasoning. The first refers to the ability to combine units of meaning, like words or phrases, into larger constructs, such as sentences or paragraphs. Entailment, on the other hand, refers to the notion of inference. A linguistic element A (e.g. a word or phrase) is said to en- tail an element B if, assuming A is true, so is B. Word-level entailment is often referred to as lexical entailment (LE), hypernym detection, or the is a re- lation (Weeds et al., 2014; Vuli´c and Mrkši´c, 2018; Kober et al., 2021), and refers to examples such as dog entails (|=) animal and gun |= weapon. Yet entailment does not occur just between two words, and has been a long standing problem in NLP (Da- gan et al., 2005; MacCartney and Manning, 2008; Marelli et al., 2014; Nie et al., 2020). When occur- ring between two sentences, it is usually referred to as natural language inference (NLI). Although arguments have been made in favour of a more probabilistic interpretation of the task (see Pavlick and Callison-Burch (2016); Pavlick and Kwiatkowski (2019), inter alia), NLI benchmarks generally abandoned the rigid binary classification for a three way classification, usually involving a neutral or UNK label1. With a few exceptions, e.g., Baroni et al. (2012); Kartsaklis and Sadrzadeh (2016); Kober et al. (2021), NLI is still the main method adopted by the NLP community to jointly study the compositional and inferential abilities of a model. However, commonly used benchmarks frequently contain spurious statistical associations that a model can use to solve the task (Poliak et al., 2018; Dasgupta et al., 2018; McCoy et al., 2019). These cues might be as simple as the presence of negation or lexical overlap (Dasgupta et al., 2018), but can be more complex, and exploit similar syn- tactic substructures between premise and hypothe- sis (McCoy et al., 2019). Popular alternatives to training and testing mod- els on datasets containing significant biases are prompting …

Content truncated to 500 words