Did the Models Understand Documents? Benchmarking Models for Language Understanding in Document-Level Relation Extraction

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6418–6435 July 9-14, 2023 ©2023 Association for Computational Linguistics Did the Models Understand Documents? Benchmarking Models for Language Understanding in Document-Level Relation Extraction Haotian Chen, Bingsheng Chen and Xiangdong Zhou School of Computer Science, Fudan University Shanghai Key Laboratory of Data Science htchen18@fudan.edu.cn, bschen21@m.fudan.edu.cn, xdzhou@fudan.edu.cn Abstract Document-level relation extraction (DocRE) at- tracts more research interest recently. While models achieve consistent performance gains in DocRE, their underlying decision rules are still understudied: Do they make the right predic- tions according to rationales? In this paper, we take the first step toward answering this ques- tion and then introduce a new perspective on comprehensively evaluating a model. Specifi- cally, we first conduct annotations to provide the rationales considered by humans in DocRE. Then, we conduct investigations and reveal the fact that: In contrast to humans, the representa- tive state-of-the-art (SOTA) models in DocRE exhibit different decision rules. Through our proposed RE-specific attacks, we next demon- strate that the significant discrepancy in deci- sion rules between models and humans severely damages the robustness of models and renders them inapplicable to real-world RE scenarios. After that, we introduce mean average preci- sion (MAP) to evaluate the understanding and reasoning capabilities of models. According to the extensive experimental results, we finally appeal to future work to consider evaluating both performance and the understanding ability of models for the development of their appli- cations. We make our annotations and code publicly available1. 1 Introduction Relation extraction (RE), aiming to extract rela- tions between entities from texts, plays an impor- tant role in constructing a large-scale knowledge graph (Riedel et al., 2010; Hendrickx et al., 2010). Most previous work extract relations from a single sentence (Zelenko et al., 2002; Wei et al., 2020; Shang et al., 2022), while recent studies adopt mul- tiple sentences as a whole to harvest more relations including inter-sentence relations (Yao et al., 2019), i.e., document-level relation extraction (DocRE). 1https://github.com/Hytn/DocRED-HWE Figure 1: An example from DocRED. DocRE is more challenging because models are required to synthesize all information of a given document and then predict relations by reasoning and language understanding (Yao et al., 2019; Nan et al., 2020; Zeng et al., 2020). Previous work obtains consistent performance gains on DocRED (Yao et al., 2019), the proposal of which has benefited the rapid development of DocRE (Huang et al., 2022). However, the extent to which their proposed methods possess language understanding and reasoning capabilities is still un- derstudied. A common evaluation method is to measure average error across a test set, which ne- glects the situations where models can make right predictions according to wrong features. As shown in Figure 1, the model accurately predicts the rela- tion between Vˇera ˇCáslavská and Czech as humans do. However, the evidence words considered by models are incomprehensible to humans. Simi- lar situations, where models improve their perfor- mance by recognizing the spurious patterns, are identified by parts of …

Content truncated to 500 words