Factoring Statutory Reasoning as Language Understanding Challenges

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2742–2758 August 1–6, 2021. ©2021 Association for Computational Linguistics 2742 Factoring Statutory Reasoning as Language Understanding Challenges Nils Holzenberger and Benjamin Van Durme Center for Language and Speech Processing Johns Hopkins University Baltimore, Maryland, USA nilsh@jhu.edu vandurme@cs.jhu.edu Abstract Statutory reasoning is the task of determin- ing whether a legal statute, stated in natural language, applies to the text description of a case. Prior work introduced a resource that ap- proached statutory reasoning as a monolithic textual entailment problem, with neural base- lines performing nearly at-chance. To address this challenge, we decompose statutory reason- ing into four types of language-understanding challenge problems, through the introduction of concepts and structure found in Prolog pro- grams. Augmenting an existing benchmark, we provide annotations for the four tasks, and baselines for three of them. Models for statu- tory reasoning are shown to benefit from the additional structure, improving on prior base- lines. Further, the decomposition into subtasks facilitates finer-grained model diagnostics and clearer incremental progress. 1 Introduction As more data becomes available, Natural Language Processing (NLP) techniques are increasingly be- ing applied to the legal domain, including for the prediction of case outcomes (Xiao et al., 2018; Vacek et al., 2019; Chalkidis et al., 2019a). In the US, cases are decided based on previous case outcomes, but also on the legal statutes compiled in the US code. For our purposes, a case is a set of facts described in natural language, as in Figure 1, in blue. The US code is a set of docu- ments called statutes, themselves decomposed into subsections. Taken together, subsections can be viewed as a body of interdependent rules specified in natural language, prescribing how case outcomes are to be determined. Statutory reasoning is the task of determining whether a given subsection of a statute applies to a given case, where both are expressed in natural language. Subsections are im- plicitly framed as predicates, which may be true or false of a given case. Holzenberger et al. (2020) in- troduced SARA, a benchmark for the task of statu- tory reasoning, as well as two different approaches to solving this problem. First, a manually-crafted symbolic reasoner based on Prolog is shown to per- fectly solve the task, at the expense of experts writ- ing the Prolog code and translating the natural lan- guage case descriptions into Prolog-understandable facts. The second approach is based on statistical machine learning models. While these models can be induced computationally, they perform poorly because the complexity of the task far surpasses the amount of training data available. We posit that statutory reasoning as presented to statistical models is underspecified, in that it was cast as Recognizing Textual Entailment (Da- gan et al., 2005) and linear regression. Taking inspiration from the structure of Prolog programs, we re-frame statutory reasoning as a sequence of four tasks, prompting us to introduce a novel ex- tension …

Content truncated to 500 words