Nibbling at the Hard Core of Word Sense Disambiguation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4724 - 4737 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Nibbling at the Hard Core of Word Sense Disambiguation Marco Maru1, Simone Conia1, Michele Bevilacqua1, and Roberto Navigli2 Sapienza NLP Group 1Department of Computer Science 2Department of Computer, Control and Management Engineering Sapienza University of Rome firstname.lastname@uniroma1.it Abstract With state-of-the-art systems having finally attained estimated human performance, Word Sense Disambiguation (WSD) has now joined the array of Natural Language Processing tasks that have seemingly been solved, thanks to the vast amounts of knowledge encoded into Transformer-based pre-trained language models. And yet, if we look below the surface of raw figures, it is easy to realize that current approaches still make trivial mistakes that a human would never make. In this work, we provide evidence showing why the F1 score metric should not simply be taken at face value and present an exhaustive analysis of the errors that seven of the most representative state-of-the-art systems for English all-words WSD make on traditional evaluation bench- marks. In addition, we produce and release a collection of test sets featuring (a) an amended version of the standard evaluation benchmark that fixes its lexical and semantic inaccuracies, (b) 42D, a challenge set devised to assess the resilience of systems with respect to least frequent word senses and senses not seen at training time, and (c) hardEN, a challenge set made up solely of instances which none of the investigated state-of-the-art systems can solve. We make all of the test sets and model predictions available to the research community at https://github.com/ SapienzaNLP/wsd-hard-benchmark. 1 Introduction In recent years, Natural Language Processing (NLP) has witnessed a quantum leap in bench- mark task performance, mainly thanks to the adop- tion of two major technical innovations: the Trans- former architecture (Vaswani et al., 2017) and trans- fer learning from language models pre-trained on massive amounts of textual data (Devlin et al., 2019; Lewis et al., 2020). The impact of these breakthroughs was so strong that, on many bench- marks, the performance of human non-experts was surpassed (Wang et al., 2019b), prompting re- searchers to release new, more challenging bench- marks (Wang et al., 2019a). Word Sense Disambiguation (WSD), the task of automatically assigning a meaning to an ambigu- ous word in context (Bevilacqua et al., 2021), is un- dergoing a similar process: current state-of-the-art systems are now capable of attaining and surpass- ing the F1 score of 80%1 on standard test datasets (Bevilacqua and Navigli, 2020; Barba et al., 2021a; Conia and Navigli, 2021; Kohli, 2021), a figure of- ten reported as the estimated human performance, because it corresponds to the highest recorded inter-annotator agreement (Edmonds and Kilgarriff, 2002; Navigli et al., 2007; Palmer et al., 2007). Matching and/or surpassing human performance reasonably triggers the assumption that systems are capable of carrying out tasks in real-world scenar- ios as effectively as their human counterparts (Kiela et al., 2021), to the point where …

Content truncated to 500 words