Continuous Entailment Patterns for Lexical Inference in Context

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6952–6959 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6952 Continuous Entailment Patterns for Lexical Inference in Context Martin Schmitt and Hinrich Schütze Center for Information and Language Processing (CIS) LMU Munich, Germany martin@cis.lmu.de Abstract Combining a pretrained language model (PLM) with textual patterns has been shown to help in both zero- and few-shot settings. For zero-shot performance, it makes sense to design patterns that closely resemble the text seen during self- supervised pretraining because the model has never seen anything else. Supervised training allows for more flexibility. If we allow for to- kens outside the PLM’s vocabulary, patterns can be adapted more flexibly to a PLM’s id- iosyncrasies. Contrasting patterns where a “to- ken” can be any continuous vector vs. those where a discrete choice between vocabulary el- ements has to be made, we call our method CONtinuous pAtterNs (CONAN). We evaluate CONAN on two established benchmarks for lex- ical inference in context (LIiC) a.k.a. predicate entailment, a challenging natural language un- derstanding task with relatively small training sets. In a direct comparison with discrete pat- terns, CONAN consistently leads to improved performance, setting a new state of the art. Our experiments give valuable insights into the kind of pattern that enhances a PLM’s performance on LIiC and raise important questions regard- ing our understanding of PLMs using text pat- terns.1 1 Introduction Lexical inference in context (LIiC) – also called predicate entailment – is a variant of natural lan- guage inference (NLI) or recognizing textual entail- ment (Dagan et al., 2013) with focus on the lexical semantics of verbs and verbal expressions (Levy and Dagan, 2016; Schmitt and Schütze, 2019). Its goal is to detect entailment between two very sim- ilar sentences, i.e., sentences that share subject and object and only differ in the predicate, e.g., PERSON(A) runs ORG(B) → PERSON(A) leads ORG(B). NLI models that were not specifically 1Our code is publicly available: https://github. com/mnschmit/conan trained with lexical knowledge have been reported to struggle with this task (Glockner et al., 2018; Schmitt and Schütze, 2019), making LIiC an im- portant evaluation criterion for general language understanding. Other use cases for this kind of lexical entailment knowledge include question an- swering (Schoenmackers et al., 2010; McKenna et al., 2021), event coreference (Shwartz et al., 2017; Meged et al., 2020), and link prediction in knowledge graphs (Hosseini et al., 2019). Although LIiC is an inherently directional task, symmetric cosine similarity in a vector space, such as word2vec (an, 2013), has long been the state of the art for this task. Only recently transfer learn- ing with pretrained Transformer (Vaswani et al., 2017) language models (Devlin et al., 2019), has led to large improvements for LIiC. Schmitt and Schütze (2021) combine natural language (NL) pat- terns with a pretrained language model (PLM) and not only set a new state of the art but also beat baselines without access to such patterns. Empirical findings suggest that a good …

Content truncated to 500 words