Do Transformer Networks Improve the Discovery of Rules from Text?

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3706–3714 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3706 Do Transformer Networks Improve the Discovery of Rules from Text? Mahdi Rahimi, Mihai Surdeanu Department of Computer Science University of Arizona, Tucson, Arizona, USA {marahimi, msurdeanu}@email.arizona.edu Abstract With their Discovery of Inference Rules from Text (DIRT) algorithm, Lin and Pantel (2001) made a seminal contribution to the field of rule acquisition from text, by adapting the distributional hypothesis of Harris (1954) to patterns that model binary relations such as X treat Y, where patterns are implemented as syntactic dependency paths. DIRT’s relevance is renewed in today’s neural era given the recent focus on interpretability in the field of natural language processing. We propose a novel take on the DIRT algorithm, where we implement the distributional hypothesis using the contextualized embeddings provided by BERT, a transformer-network-based language model (Vaswani et al., 2017; Devlin et al., 2018). In particular, we change the similarity measure between pairs of slots (i.e., the set of words matched by a pattern) from the original formula that relies on lexical items to a formula computed using contextualized embeddings. We empirically demonstrate that this new similarity method yields a better implementation of the distributional hypothesis, and this, in turn, yields patterns that outperform the original algorithm in the question answering-based evaluation proposed by Lin and Pantel (2001). Keywords: Rule Acquisition, Distributional Hypothesis, DIRT 1. Introduction Lin and Pantel (2001) proposed a method for the ac- quisition of rules from text through an extension of the distributional hypothesis (Harris, 1954) to rules.1 That is, “if two paths tend to link the same set of words, . . . their meanings are similar.” The resulting algorithm, called Discovery of Inference Rules from Text (DIRT), has had a wide impact in natural language processing (NLP). For example, it has been shown that such rules acquisition improves question answer- ing (Ravichandran and Hovy, 2002), information ex- traction (Shinyama and Sekine, 2006), and textual en- tailment (Melamud et al., 2013b). For example, the pat- tern X treat Y may be used to extract the answer to the question Which drugs relieve stomach ache? (Mela- mud et al., 2013b). The algorithm itself has been im- proved and adapted to various scenarios (see the Re- lated Work section for a larger discussion). Unfortunately, the idea of using such rule acquisi- tion and the DIRT algorithm have largely been forgot- ten once the “deep learning tsunami” hit NLP around 2015 (Manning, 2015). While neural network have brought unquestionable performance improvements to most NLP tasks (as one example among many, the leader board for TACRED,2 a popular relation extrac- tion task, is completely dominated by neural methods), some advantages of rule-based approaches have been lost. Most importantly, rule-based models are inter- pretable. That is, every model decision can be assigned to one or a small number of rules, which, generally, are understandable by humans. Moreover, unlike post-hoc explainability methods …

Content truncated to 500 words