Testing Focus and Non-at-issue Frameworks with a Question-under-Discussion-Annotated Corpus

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 524–533 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 524 VALET: Rule-Based Information Extraction for Rapid Deployment Dayne Freitag, John Cadigan, Robert Sasseen, Paul Kalmar SRI International 9988 Hibert Street, Suite 203 San Diego, California 92131, USA firstname.lastname@sri.com Abstract We present VALET, a framework for rule-based information extraction written in Python. VALET departs from legacy approaches predicated on cascading finite-state transducers, instead offering direct support for mixing heterogeneous information—lexical, orthographic, syntactic, corpus-analytic—in a succinct syntax that supports context-free idioms. We show how a handful of rules suffices to implement sophisticated matching, and describe a user interface that facilitates exploration for development and maintenance of rule sets. Arguing that rule-based information extraction is an important methodology early in the development cycle, we describe an experiment in which a VALET model is used to annotate examples for a machine learning extraction model. While learning to emulate the extraction rules, the resulting model generalizes them, recognizing valid extraction targets the rules failed to detect. Keywords: information extraction, rule-based methods 1. Introduction Under the current state of the art, achieving peak ac- curacy in sentence-level information extraction (IE) in- volves training supervised models on substantial quan- tities of text annotated at the phrase level. Whether the objective is named entity or concept recognition, re- lation extraction, event extraction, or some slot-filling endeavor, the basic recipe is the same: extraction ele- ments must be identified and labeled in context, their boundaries marked, and phrasal elements optionally drawn into typed relations with each other. Once enough examples have been annotated in this fash- ion, the literature describes a large number of machine learning models that can be trained to perform extrac- tion at accuracies ranging from strong to adequate, e.g., (Ma and Hovy, 2016; Zhang et al., 2018; Wadden et al., 2019). To the extent that an extraction problem is nuanced, e.g. exhibits high linguistic variability or requires supra-lexical interpretation, data requirements are increased, necessitating the employment of a team of annotators, often governed by detailed guidelines to maximize annotation consistency. Thus, the IE devel- opment process can be quite expensive, often too ex- pensive for many applications that might benefit from special-purpose extraction. The models that result from this process are also chron- ically limited. Over the decades during which IE has been a focus of study, the field has emphasized prob- lem formulations of broad applicability. For exam- ple, recognizing mentions of named entities and cer- tain concepts, such as dates or monetary amounts, is a necessary component of many applications of text un- derstanding. However, all such work must commit to specific genres (e.g., newswire) and levels of granular- ity, necessarily limiting the applicability of any result- ing model. Generally, it is not difficult to find prac- tical problems of text understanding for which use of such community models is problematic, whether be- cause the formalization assumed by the model is an awkward fit to …

Content truncated to 500 words