Passing a USA National Bar Exam: a First Corpus for Experimentation

Passing a USA National Bar Exam: a First Corpus for Experimentation Biralatei Fawei, Adam Wyner, and Jeff Pan Department of Computing Science, University of Aberdeen, Aberdeen, United Kingdom biralatei.fawei,azwyner,jeff.z.pan@abdn.ac.uk Abstract Bar exams provide a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides an excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper provides a corpus and experimental results with material derived from a real bar exam, treating the problem as a form of textual entailment from the question to an answer. The providers of the bar exam material set the Gold Standard, which is the answer key. The experiments carried out using the ‘out of the box’ the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, but it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future. The corpus facilitates experimentation by other researchers. Keywords: textual entailment, bar exam, legal reasoning, natural language processing 1. Introduction Bar exams, which are extensive in-depth examinations about legal information and reasoning, provide a key water- shed by which legal professionals demonstrate their knowl- edge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction and topic, and it validates the examinee’s knowledge of the law. We can say, then, that the bar exam encapsulates a range of legal knowledge. Thus, the bar provides an excellent benchmark for the performance of legal information systems which at- tempt to represent and reason with the law, since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. This paper presents a dataset and a first experiment with bar exam material derived from the United States Multi- state Bar Examination (MBE) material, provided by the Na- tional Conference of Bar Examiners (NCBE) 1. The Gold Standard (the correct answers) is provided by the NCBE. The paper reports a textual entailment study on this US bar exam material, running the Excitement Open Platform (EOP)2 for textual entailment ‘out of the box’ (Dagan et al., 2009). In the experiment, we treat the the relationship between the question and the multiple-choice answers as a form of textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, though it performs poorly in identify- ing the correct answer (entailment). The results provide a baseline performance measure against which to evaluate fu- ture improvements. The …

Content truncated to 500 words