Addressing the MFS Bias in WSD systems

Addressing the MFS Bias in WSD Systems Marten Postma*, Ruben Izquierdo*, Eneko Agirre†, German Rigau†, Piek Vossen* Vrije Universiteit Amsterdam*; University of the Basque Country† m.c.postma@vu.nl, ruben.izquierdobevia@vu.nl, e.agirre@ehu.eus, german.rigau@ehu.eus, piek.vossen@vu.nl Abstract Word Sense Disambiguation (WSD) systems tend to have a strong bias towards assigning the Most Frequent Sense (MFS), which results in high performance on the MFS but in a very low performance on the less frequent senses. We addressed the MFS bias in WSD systems by combining the output from a WSD system with a set of mostly static features to create a MFS classifier to decide when to and not to choose the MFS. The output from this MFS classifier, which is based on the Random Forest algorithm, is then used to modify the output from the original WSD system. We applied our classifier to one of the state-of-the-art super- vised WSD systems, i.e. IMS, and to of the best state-of-the-art unsupervised WSD systems, i.e. UKB. Our main finding is that we are able to improve the system output in terms of choosing between the MFS and the less frequent senses. When we apply the MFS classifier to fine-grained WSD, we observe an improvement on the less frequent sense cases, whereas we maintain the overall recall. Keywords: WSD, MFS, Random Forest 1. Introduction Word Sense Disambiguation (WSD) is generally defined as a classification task. The goal is to determine which sense of a word or multi-word expression is used in a linguistic context (Agirre and Edmonds, 2007). In order to be able to compare the performance of the techniques that have been applied to try to solve this task, WSD evaluation tasks have been organized. Izquierdo (2015) conducted an error anal- ysis of five all-words tasks to define the problem space of WSD systems.1 The most striking results came from the analysis of compar- ing the average recall of WSD approaches on instances in which the Most Frequent Sense (MFS) was the gold sense versus when one of the less frequent senses (LFS) was the gold sense (Figure 1). We observe a clear trend that sys- tems excel on MFS cases, exceeding the average recall by far, whereas the recall drops dramatically for all LFS cases, far below the average recall. The average performance of all systems is therefore largely influenced by a very skewed and limited capacity to recognize senses in contexts. The question we address here in this paper is: how can we in- crease the performance for the LFS part of the task? The poor recall of WSD systems on the LFS can be ex- plained by at least two observations. Firstly, the systems are biased towards the MFS sense, which is why they opt for the MFS when one of the LFS applies. Secondly in those cases where the WSD systems are correctly distin- guishing between the MFS and the LFS, they opt for the wrong sense from the set of the LFS, probably due to lack of training data for the LFS. We …

Content truncated to 500 words