From Examples to Rules: Neural Guided Rule Synthesis for Information Extraction

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6180–6189 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6180 From Examples to Rules: Neural Guided Rule Synthesis for Information Extraction Robert Vacareanu, Marco A. Valenzuela-Escárcega, George C. G. Barbosa, Rebecca Sharp, Gus Hahn-Powell, Mihai Surdeanu University of Arizona Tucson, AZ, USA {rvacareanu, gcgbarbosa, hahnpowell, msurdeanu}@email.arizona.edu {marcovalenzuelaescarcega, bsharpataz}@gmail.com Abstract While deep learning approaches to information extraction have had many successes, they can be difficult to augment or maintain as needs shift. Rule-based methods, on the other hand, can be more easily modified. However, crafting rules requires expertise in linguistics and the domain of interest, making it infeasible for most users. Here we attempt to combine the advantages of these two directions while mitigating their drawbacks. We adapt recent advances from the adjacent field of program synthesis to information extraction, synthesizing rules from provided examples. We use a transformer-based architecture to guide an enumerative search, and show that this reduces the number of steps that need to be explored before a rule is found. Further, we show that without training the synthesis algorithm on the specific domain, our synthesized rules achieve state-of-the-art performance on the 1-shot scenario of a task that focuses on few-shot learning for relation classification, and competitive performance in the 5-shot scenario. Keywords: rule-based information extraction, rule synthesis 1. Introduction The “deep learning tsunami” that “hit” natural language processing (NLP) (Manning, 2015) has brought tremendous improvements in performance to most NLP applications. However, these bene- fits do not come for free. One drawback of deep learning is its opacity, which limits the ability for users to make incremental improvements to de- ployed systems. In particular, the entanglement of deep learning approaches means that “changing one thing changes everything” (Sculley et al., 2015). In contrast, rule-based approaches are much more amenable to incremental improvements as each indi- vidual rule can be interpreted explicitly and unam- biguously. This is critical for systems which will be deployed and maintained for long periods of time. Further, because rules encode expert knowledge, experts can write them without first curating a large number of examples. However, an important drawback to rule-based ap- proaches is that rule development is time consum- ing, and requires expertise in both the domain at hand and in linguistics. To mitigate this limitation, many directions before the “deep learning tsunami” focused on rule learning from examples (Yarowsky, 1995; Riloff, 1996; Collins and Singer, 1999; Abney, 2002; McIntosh, 2010; Gupta and Manning, 2014, inter alia). Here we propose a novel method for rule synthesis from examples that combines the strengths of deep learning with the advantages of rule-based meth- ods. By utilizing a self-supervised pre-trained trans- former, we minimize the number of examples needed from the expert. By generating human-readable rules, the resulting grammar can be adjusted or extended incrementally as needs shift. Our ability to generate rules from limited examples is key to our approach as this mimics a real-world setting, where …

Content truncated to 500 words