Improving negation detection with negation-focused pre-training

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4188 - 4193 July 10-15, 2022 ©2022 Association for Computational Linguistics Improving negation detection with negation-focused pre-training Hung Thinh Truong1 Timothy Baldwin1,3 Trevor Cohn1 Karin Verspoor2 1The University of Melbourne, 2RMIT University, 3MBZUAI hungthinht@student.unimelb.edu.au, tb@ldwin.net, trevor.cohn@unimelb.edu.au, karin.verspoor@rmit.edu.au Abstract Negation is a common linguistic feature that is crucial in many language understanding tasks, yet it remains a hard problem due to diversity in its expression in different types of text. Re- cent work has shown that state-of-the-art NLP models underperform on samples containing negation in various tasks, and that negation detection models do not transfer well across domains. We propose a new negation-focused pre-training strategy, involving targeted data augmentation and negation masking, to better incorporate negation information into language models. Extensive experiments on common benchmarks show that our proposed approach improves negation detection performance and generalizability over the strong baseline Neg- BERT (Khandelwal and Sawant, 2020). 1 Introduction Negation is an important linguistic phenomenon that appears commonly in natural language but is underrepresented in common NLP benchmarks (Hossain et al., 2020). Furthermore, the Checklist benchmark (Ribeiro et al., 2020) shows that most sentiment analyzers and machine comprehension models struggle with samples containing negation. Negation is even more important in biomedical domain text, where patients are carefully defined as having/not having specific characteristics. Even within the biomedical domain, there are many types of text such as clinical notes, lab reports, or re- search publications, each with particular character- istics in relation to the use of negation. A recent study on English texts found that negation detection models do not transfer well across domains, due to variations in expression of negation (Khandelwal and Sawant, 2020). It remains a challenge to solve negation in general, even with state-of-the-art NLP models. Negation detection is typically defined as con- sisting of the two sub-tasks of: (1) cue detection, detecting the cue phrase that triggers the negation; and (2) scope resolution, determining the affected spans that are negated. There are three primary datasets that have been used to evaluate negation: (1) the BioScope corpus (Vincze et al., 2008) in- cludes full papers and abstracts of biological pa- pers; (2) the SFU corpus (Konstantinova et al., 2012) is a collection of product reviews; and (3) the Sherlock dataset (Morante and Blanco, 2012) con- sists of short literary works. There are differences in annotation schemes across the datasets, such as whether or not the cues are included inside scope annotation, and sub-optimal cross-dataset results have been observed, providing clear indications that the datasets are highly divergent in language use and negation types. In this work, we aim to extend the transfer learning capability of NegBERT (Khandelwal and Sawant, 2020) through additional pre-training with task-related augmented training data, and a new masking objective. Our contributions are: • We introduce an approach to augmenting data to emphasize negation in pre-training. • We propose a novel extension to the standard random masked …

Content truncated to 500 words