Cross-domain Generalization for AMR Parsing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1120 - 1138 December 7-11, 2022 ©2022 Association for Computational Linguistics Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts Santosh T.Y.S.S1∗ and Shanshan Xu1∗ and Oana Ichim2 and Matthias Grabmair1 1School of Computation, Information, and Technology; Technical University of Munich, Germany 2Graduate Institute of International and Development Studies, Geneva, Switzerland {santosh.tokala, shanshan.xu, matthias.grabmair}@tum.de oana.ichim@graduateinstitute.ch Abstract This work demonstrates that Legal Judgement Prediction systems without expert-informed ad- justments can be vulnerable to shallow, distract- ing surface signals that arise from corpus con- struction, case distribution, and confounding factors. To mitigate this, we use domain ex- pertise to strategically identify statistically pre- dictive but legally irrelevant information. We adopt adversarial training to prevent the sys- tem from relying on it. We evaluate our de- confounded models by employing interpretabil- ity techniques and comparing to expert annota- tions. Quantitative experiments and qualitative analysis show that our deconfounded model consistently aligns better with expert rationales than baselines trained for prediction only. We further contribute a set of reference expert an- notations to the validation and testing partitions of an existing benchmark dataset of European Court of Human Rights cases. 1 Introduction The task of Legal Judgment Prediction (LJP) has recently gained increasing attention in the legal and mainstream NLP communities (Aletras et al., 2016; Zhong et al., 2018; Medvedeva et al., 2020; Liu et al., 2019; Sert et al., 2021). Legal cases are resolved through the exchange of arguments in front of a decision body by lawyers who rep- resent litigating parties. This typically involves evidential reasoning, the determination of relevant rules from sources of law (e.g., codes, regulations, precedent), their application to the case, and the bal- ancing of legal and societal values. In the NLP con- text, LJP takes the form of classifying the outcome of a case from some textual representation of its specific facts, effectively skipping legal reasoning. This forms a counterpoint to knowledge-focused approaches to outcome prediction (e.g., Brüning- haus and Ashley, 2005; Branting, 2013; Grabmair, *These authors contributed equally to this work Spurious Tokens ECtHR Decision Fact Statement Case Outcome Decision Tree Classifiers Spurious Attributes Respondent State Doc. Length Legal Expert Review Hierarchical BERT Classifier Adversarial Decounfounder Evaluation IG Focus Map Expert Rationale Figure 1: Our deconfounding experiment architecture 2017) that connect to a lawyer’s understanding of the domain but also require substantial knowledge engineering. This carries particular risk in the legal domain, where systems may rely on data elements that are statistically predictive but legally irrelevant, or even forbidden as decision criteria (e.g., the race of an accused person). This can lead to undesirable con- sequences, ranging from suboptimal litigation strat- egy decisions, flawed inference about factors pre- dictive for the outcome, to disparate impact of de- cisions across groups that are to be treated equally. If legal decisions are to be informed by predictive systems processing textual case descriptions, then such systems must strive to …

Content truncated to 500 words