Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99–110, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison Alessandro Raganato, Jose Camacho-Collados and Roberto Navigli Department of Computer Science Sapienza University of Rome {raganato,collados,navigli}@di.uniroma1.it Abstract Word Sense Disambiguation is a long- standing task in Natural Language Pro- cessing, lying at the core of human lan- guage understanding. However, the evalu- ation of automatic systems has been prob- lematic, mainly due to the lack of a re- liable evaluation framework. In this pa- per we develop a unified evaluation frame- work and analyze the performance of various Word Sense Disambiguation sys- tems in a fair setup. The results show that supervised systems clearly outper- form knowledge-based models. Among the supervised systems, a linear classi- fier trained on conventional local features still proves to be a hard baseline to beat. Nonetheless, recent approaches exploit- ing neural networks on unlabeled corpora achieve promising results, surpassing this hard baseline in most test sets. 1 Introduction Word Sense Disambiguation (WSD) has been a long-standing task in Natural Language Process- ing (NLP). It lies at the core of language under- standing and has already been studied from many different angles (Navigli, 2009; Navigli, 2012). However, the field seems to be slowing down due to the lack of groundbreaking improvements and the difficulty of integrating current WSD sys- tems into downstream NLP applications (de La- calle and Agirre, 2015). In general the field does not have a clear path, partially owing to the fact that identifying real improvements over existing approaches becomes a hard task with current eval- uation benchmarks. This is mainly due to the lack of a unified framework, which prevents di- rect and fair comparison among systems. Even though many evaluation datasets have been con- structed for the task (Edmonds and Cotton, 2001; Snyder and Palmer, 2004; Navigli et al., 2007; Pradhan et al., 2007; Agirre et al., 2010a; Nav- igli et al., 2013; Moro and Navigli, 2015, in- ter alia), they tend to differ in format, construc- tion guidelines and underlying sense inventory. In the case of the datasets annotated using WordNet (Miller, 1995), the de facto sense inventory for WSD, we encounter the additional barrier of hav- ing text annotated with different versions. These divergences are in the main solved individually by using or constructing automatic mappings. The quality check of such mapping, however, tends to be impractical and this leads to mapping errors which give rise to additional system inconsisten- cies in the experimental setting. This issue is di- rectly extensible to the training corpora used by supervised systems. In fact, results obtained by supervised or semi-supervised systems reported in the literature are not completely reliable, because the systems may not necessarily have been trained on the same corpus, or the corpus was prepro- cessed differently, or annotated with a sense inven- tory different from the test data. …

Content truncated to 500 words