Negation-Instance Based Evaluation of End-to-End Negation Resolution

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 528–543 November 10–11, 2021. ©2021 Association for Computational Linguistics 528 Negation-Instance Based Evaluation of End-to-End Negation Resolution Elizaveta Sineva1,2, Stefan Grünewald1,2, Annemarie Friedrich1, Jonas Kuhn2 1Bosch Center for Artificial Intelligence, Renningen, Germany 2Institut für Maschinelle Sprachverarbeitung, University of Stuttgart elisavetas2106@gmail.com stefan.gruenewald|annemarie.friedrich@de.bosch.com jonas.kuhn@ims.uni-stuttgart.de Abstract In this paper, we revisit the task of negation resolution, which includes the subtasks of cue detection (e.g. “not”, “never”) and scope reso- lution. In the context of previous shared tasks, a variety of evaluation metrics have been pro- posed. Subsequent works usually use differ- ent subsets of these, including variations and custom implementations, rendering meaning- ful comparisons between systems difficult. Ex- amining the problem both from a linguistic per- spective and from a downstream viewpoint, we here argue for a negation-instance based ap- proach to evaluating negation resolution. Our proposed metrics correspond to expectations over per-instance scores and hence are intu- itively interpretable. To render research com- parable and to foster future work, we provide results for a set of current state-of-the-art sys- tems for negation resolution on three English corpora, and make our implementation of the evaluation scripts publicly available. 1 Introduction Negation is a complex semantic phenomenon in natural language that “transforms an expression into another expression whose meaning is in some way opposed to the original” (Morante and Blanco, 2021). It occurs frequently, with the proportion of sentences with negation in English corpora rang- ing between 9 and 32% (Jiménez-Zafra et al., 2020b). Natural Language Processing (NLP) ap- plications that may benefit from negation resolu- tion include sentiment analysis (Wiegand et al., 2010; Moore and Barnes, 2021) and information extraction. Negation is also still a challenge in machine translation (Fancellu and Webber, 2015; Bentivogli et al., 2016; Hossain et al., 2020a) and natural language inference (Hossain et al., 2020c; Geiger et al., 2020). Negation Resolution (Morante and Blanco, 2021) refers to the task of automatically retriev- ing the elements of a sentence that are affected by Test Sentences (gold scope) ....... System... A System B TP / FP / FN TP / FP / FN (1) ..If not, .I .. ’ll .... have to do with you. 0 / 4 / 0 0 / 0 / 0   (2) He made no ....... remark, but the 1 / 0 / 2 3 / 0 / 0 matter remained in his thoughts.   (3) Well, Mrs. Warren, I can see that 16 / 0 / 0 10 / 2 / 6 you have any particular cause  () for concern, nor ... do .I ........... understand .... why .I, ....... whose ..... time ..is ..of....... some ...... value, ....... should ......... interfere ..in.... the....... matter. Scope Tokens P/R: 81.0/89.5 86.7/68.4 F1: 85.0 76.5 Instance-Based P/R: 66.7/77.8 94.4/87.5 F1: 71.8 90.8 Figure 1: Different scoring metrics for negation scope resolution predictions. / = our judgment of system correctness taking into account linguistic crite- ria. Parentheses indicate partial correctness. the negation introduced …

Content truncated to 500 words