Seeking Clozure: Robust Hypernym extraction from BERT with Anchored Prompts

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 193–206 July 13-14, 2023 ©2023 Association for Computational Linguistics Seeking Clozure: Robust Hypernym Extraction from BERT with Anchored Prompts Chunhua Liu Trevor Cohn∗ Lea Frermann School of Computing and Information Systems The University of Melbourne chunhua@student.unimelb.edu.au {tcohn,lfrermann}@unimelb.edu.au Abstract The automatic extraction of hypernym knowl- edge from large language models like BERT is an open problem, and it is unclear whether methods fail due to a lack of knowledge in the model or shortcomings of the extraction meth- ods. In particular, methods fail on challenging cases which include rare or abstract concepts, and perform inconsistently under paraphrased prompts. In this study, we revisit the long line of work on pattern-based hypernym extraction, and use it as a diagnostic tool to thoroughly examine the hypernomy knowledge encoded in BERT and the limitations of hypernym extrac- tion methods. We propose to construct prompts from established pattern structures: definitional (X is a Y); lexico-syntactic (Y such as X); and their anchored versions (Y such as X or Z). We devise an automatic method for anchor pre- diction, and compare different patterns in: (i) their effectiveness for hypernym retrieval from BERT across six English data sets; (ii) on chal- lenge sets of rare and abstract concepts; and (iii) on consistency under paraphrasing. We show that anchoring is particularly useful for abstract concepts and in enhancing consistency across paraphrases, demonstrating how estab- lished methods in the field can inform prompt engineering.1 1 Introduction Semantic relations play a central role in knowl- edge representation (Miller, 1995) and taxonomy construction (Snow et al., 2006; Navigli et al., 2011). As the backbone of semantic relations, hy- ponymy/hypernymy relations express a hierarchi- cal relation between a specific concept (the hy- ponym; e.g., dog) and a general one (the hyper- nym; e.g., mammal), and form the foundation of human concept understanding (Yu et al., 2015) and relation reasoning (Lyons, 1977; Green et al., ∗Now at Google DeepMind. 1Code and test sets are available at https://github. com/ChunhuaLiu596/AnchoredPrompts Lexico-Syntactic Prompt __ such as keas. Anchored Prompt __ such as keas and parrots. fish birds Definitional Prompt A kea is a __. fish Figure 1: Example prompts for hypernym prediction, derived from established pattern structures. 2002). Given its fundamental role, the automatic extraction of hypernym knowledge from large texts (Hearst, 1992; Roller et al., 2018) or pre-trained lan- guage models (PLMs) (Takeoka et al., 2021; Jain and Espinosa Anke, 2022), and its injection into NLP methods are active areas of research (Peters et al., 2019). The unsupervised extraction of hypernyms from PLMs by prompting has attracted recent attention, e.g., using patterns like A dog is a type of [MASK] and retrieving the most likely filler words from the model (Ettinger, 2020; Weir et al., 2020; Jain and Espinosa Anke, 2022). Results were mixed: while PLMs can reliably predict hypernyms of concrete and frequent hyponyms (Ettinger, 2020; Weir et al., 2020), experiments on more challenging data sets show a quick …

Content truncated to 500 words