A Two-Stage Masked LM Method for Term Set Expansion

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6829–6835 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6829 A Two-Stage Masked LM Method for Term Set Expansion Guy Kushilevitz Technion- Israel institute of technology guykush@cs.technion.ac.il Shaul Markovitch Technion- Israel institute of technology shaulm@cs.technion.ac.il Yoav Goldberg Bar-Ilan University Allen Institute for AI yogo@cs.biu.ac.il Abstract We tackle the task of Term Set Expansion (TSE): given a small seed set of example terms from a semantic class, finding more members of that class. The task is of great practical utility, and also of theoretical util- ity as it requires generalization from few ex- amples. Previous approaches to the TSE task can be characterized as either distribu- tional or pattern-based. We harness the power of neural masked language models (MLM) and propose a novel TSE algorithm, which combines the pattern-based and distributional approaches. Due to the small size of the seed set, fine-tuning methods are not effec- tive, calling for more creative use of the MLM. The gist of the idea is to use the MLM to first mine for informative patterns with re- spect to the seed set, and then to obtain more members of the seed class by generalizing these patterns. Our method outperforms state- of-the-art TSE algorithms. Implementation is available at: https://github.com/ guykush/TermSetExpansion-MPB/ 1 Introduction Term Set expansion (TSE) is the task of ex- panding a small seed set of terms into a larger (ideally complete) set of terms that belong to the same semantic category. For example, the seed set {“orange”, “apple”} should expand into a set of fruits, while {“orange”, “blue”} into a set of colors, and {“apple”,“google”} into a set of tech companies. Beyond being of great practical utility, the TSE task is a challenging instance of a gen- eralization from few examples problem. Solving TSE requires the algorithm to: (1) identify the de- sired concept class based on few examples; and (2) identify additional members of the class. We present an effective TSE method which is based on querying large, pre-trained masked language models (MLMs). Pre-trained language models (LMs) have been shown to contain se- mantic (Tenney et al., 2019), syntactic (Goldberg, 2019; Hewitt and Manning, 2019; Linzen et al., 2016) and factual knowledge (Petroni et al., 2019), and to be great starting points for transfer-learning to new tasks via fine-tuning on few examples. However, the TSE seed sets are too small for fine-tuning, calling for a different approach. Our method uses the MLMs directly for the task they were trained for—language-modeling—by issuing word-completion queries and operating on the re- turned word distributions.1 Previous solutions to the TSE problem (also called semantic class induction) can be roughly categorized into distributional and pattern-based approaches (Shi et al., 2010). Our method can be seen as a combination of the two. The distributional approach to TSE (Hindle, 1990; Pantel and Lin, 2002; Pantel et al., 2009; Mamou et al., 2018; Mahabal et al., 2018) operates under the hypothesis that similar words appear in …

Content truncated to 500 words