Annotation Study of Japanese Judgments on Tort for Legal Judgment Prediction with Rationales

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 779–790 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 779 Annotation Study of Japanese Judgments on Tort for Legal Judgment Prediction with Rationales Hiroaki Yamada1, Takenobu Tokunaga1, Ryutaro Ohara2 Keisuke Takeshita3, Mihoko Sumida3 Tokyo Institute of Technology1, Nakamura, Tsunoda & Matsumoto2, Hitotsubashi University3 Tokyo, Japan yamada@c.titech.ac.jp, take@c.titech.ac.jp, r.ohara@ntmlo.com kei.takeshita@r.hit-u.ac.jp, m.sumida@r.hit-u.ac.jp Abstract This paper describes a comprehensive annotation study on Japanese judgment documents in civil cases. We aim to build an annotated corpus designed for Legal Judgment Prediction (LJP), especially for torts. Our annotation scheme contains annotations of whether tort is accepted by judges as well as its corresponding rationales for explainability purpose. Our annotation scheme extracts decisions and rationales at character-level. Moreover, the scheme can capture the explicit causal relation between judge’s decisions and their corresponding rationales, allowing multiple decisions in a document. To obtain high-quality annotation, we developed an annotation scheme with legal experts, and confirmed its reliability by agreement studies with Krippendorff’s alpha metric. The result of the annotation study suggests the proposed annotation scheme can produce a dataset of Japanese LJP at reasonable reliability. Keywords: Annotated corpus, Annotation scheme, Agreement study, Legal judgment prediction, Rationale extraction 1. Introduction One of the objectives in legal information processing is to provide computational aid in the legal procedures in court cases. Legal Judgment Prediction (LJP), which predicts the outcome of a court case (Figure 1), is a cru- cial task to realise such a system. The automated LJP system can help not only legal professionals but also the general public who are not legal specialists. The system allows everyone to predict and foresee the out- come of litigation when involved in legal disputes. Peo- ple can access the system wherever, whenever. Also, the anticipated cost of LJP will be much lower than that of human legal professionals. The system is ex- pected to provide broader access to justice for people who have limited or no access to justice. Although LJP has been a longstanding research topic in Artificial Intelligence, most large-scale datasets for LJP have been proposed only recently. With the increasing popularity of machine learning(ML) based approaches, various studies proposed larger datasets to train ML models. Unfortunately, the available resources are still limited to certain languages and jurisdictions. Xiao et al. (2018) proposed a dataset for Chinese Crimi- nal cases (2.6M cases), which consist of annotations on applicable laws, charges, and prison terms. Chalkidis et al. (2019) presented 11.5K cases from the Euro- pean Court of Human Rights. Their dataset was de- signed for violated article detection and case impor- tance prediction. Katz et al. (2017) constructed a dataset from 28K cases of the Supreme Court of the United States. Chalkidis et al. (2021b) proposed a col- lection of datasets for evaluating model performance across different legal tasks including LJP in English. To train and assess LJP models, it is necessary to de- velop the LJP tasks and their datasets reflecting …

Content truncated to 500 words