More is not always better: balancing sense distributions for all-words Word Sense Disambiguation

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3496–3506, Osaka, Japan, December 11-17 2016. More is not always better: balancing sense distributions for all-words Word Sense Disambiguation Marten Postma, Ruben Izquierdo Bevia, Piek Vossen Vrije Universiteit Amsterdam De Boelelaan 1105, 1081 HV Amsterdam The Netherlands m.c.postma@vu.nl, rubensanvi@gmail.com, piek.vossen@vu.nl Abstract Current Word Sense Disambiguation systems show an extremely poor performance on low fre- quent senses, which is mainly caused by the difference in sense distributions between training and test data. The main focus in tackling this problem has been on acquiring more data or se- lecting a single predominant sense and not necessarily on the meta properties of the data itself. We demonstrate that these properties, such as the volume, provenance, and balancing, play an important role with respect to system performance. In this paper, we describe a set of experi- ments to analyze these meta properties in the framework of a state-of-the-art WSD system when evaluated on the SemEval-2013 English all-words dataset. We show that volume and provenance are indeed important, but that approximating the perfect balancing of the selected training data leads to an improvement of 21 points and exceeds state-of-the-art systems by 14 points while using only simple features. We therefore conclude that unsupervised acquisition of training data should be guided by strategies aimed at matching meta properties. 1 Introduction The task of automatically selecting the meaning of a word in a linguistic context, known as Word Sense Disambiguation (WSD), has been and is one of the main challenges for NLP. Despite the huge amount of research that has been carried out to analyze and tackle this problem, it is nevertheless still con- sidered unsolved. The best performances on the SemEval-2013 task 12: “Multilingual Word Sense Disambiguation” English all-words subtask (from now on sem2013-aw) (Navigli et al., 2013) was 72.28 (Weissenborn et al., 2015)) out of competition and 64.7 in competition (Guti´errez et al., 2013), exceeding the naive Most Frequent Sense (MFS) baseline by only 10 points at most. Supervised machine learning systems have been successful for WSD and it is commonly believed that the problem of the low performance can be solved by providing more (manually annotated) training data. However, in addition to the sparseness of training data, another aspect of the problem is the Zipfian distribution of word senses (McCarthy et al., 2007). In both training data and test data, the same single sense of a word tends to heavily dominate, making the MFS not only a baseline that is difficult to beat (Kilgarriff, 2004), but also being used as a fall-back option by many systems. Given the Zipfian distribution of word senses, it comes as no surprise that WSD systems have a bias towards assigning the MFS (Preiss, 2006). Building upon this observation, Postma et al. (2016) analyzed systems participating in previous Senseval and SemEval competitions with respect to their performance on the MFS and on all other senses, which are called the less frequent senses (LFS). This study …

Content truncated to 500 words