PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation

Findings of the Association for Computational Linguistics: EACL 2023, pages 1806–1818 May 2-6, 2023 ©2023 Association for Computational Linguistics PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation Ishan Jindala, Alexandre Rademakera, Khoi-Nguyen Trana, Huaiyu Zhua, Hiroshi Kanayamaa, Marina Danilevskya, Yunyao Lib∗ aIBM Research, bApple ishan.jindal@ibm.com, alexrad@br.ibm.com, kndtran@ibm.com, huaiyu@us.ibm.com, hkana@jp.ibm.com, mdanile@us.ibm.com, yunyaoli@apple.com Abstract Semantic role labeling (SRL) identifies predicate-argument structures in a sentence. This task is usually accomplished in four steps: predicate identification, predicate sense disam- biguation, argument identification, and argu- ment classification. Errors introduced at one step propagate to later steps. Unfortunately, the existing SRL evaluation scripts do not con- sider the full effect of this error propagation aspect. They either evaluate arguments inde- pendent of predicate sense (CoNLL09) or do not evaluate predicate sense at all (CoNLL05), yielding an inaccurate SRL model performance on the argument classification task. In this pa- per, we address key practical issues with ex- isting evaluation scripts and propose a more strict SRL evaluation metric, PriMeSRL. We observe that by employing PriMeSRL, the qual- ity evaluation of all SoTA SRL models drops significantly, and their relative rankings also change. We also show that PriMeSRL success- fully penalizes actual failures in SoTA SRL models. 1 Introduction Semantic Role Labeling (SRL) extracts predicate- argument structures from a sentence, where pred- icates represent relations (verbs, adjectives, or nouns) and arguments are the spans attached to the predicate demonstrating “who did what to whom, when, where, and how.” As one of the funda- mental natural language processing (NLP) tasks, SRL has been shown to help a wide range of NLP downstream applications such as natural lan- guage inference (Zhang et al., 2020b; Liu et al., 2022), question answering (Maqsud et al., 2014; Yih et al., 2016; Zhang et al., 2020b; Dryja´nski et al., 2022), machine translation (Shi et al., 2016; Rapp, 2022), content moderation and verification (Calvo Figueras et al., 2022; Fharook et al., 2022), information extraction (Niklaus et al., 2018; Zhang ∗b Work done while at IBM Research [Derick] broke the [window] with a [hammer] to [escape] . A0 break.01 A1 A2 AM-PRR nsubj det obj mark det obl mark obl ROOT Figure 1: An SRL example with head-based semantic roles on top of Universal Dependencies annotation. et al., 2020a). In all of these applications, the qual- ity of the underlying SRL models has a significant impact on the downstream tasks. Despite this, few studies exist on how to properly evaluate the quality of SRL systems in practice. Given a sentence, a typical SRL system obtains predicate-argument structure by following a se- ries of four steps: 1) predicate identification; 2) predicate sense disambiguation; 3) argument iden- tification; and 4) argument classification. The predicate senses and their argument labels are taken from inventories of frame definitions such as Proposition Bank (PropBank) (Palmer et al., 2005), FrameNet (Baker et al., 1998), and Verb- Net (Schuler, 2005). The accuracy of SRL extraction is affected by the correctness of each of these steps. Consider the example in Figure …

Content truncated to 500 words