MSD-1030: A Well-built Multi-Sense Evaluation Dataset for Sense Representation Models

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5802–5809 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5802 MSD-1030: A Well-built Multi-Sense Evaluation Dataset for Sense Representation Models Ting-Yu Yen1, Yang-Yin Lee1, Yow-Ting Shiue1, Hen-Hsen Huang2,3, Hsin-Hsi Chen1,3 1Department of Computer Science and Information Engineering, National Taiwan University, Taiwan 2Department of Computer Science, National Chengchi University, Taiwan 3MOST Joint Research Center for AI Technology and All Vista Healthcare, Taiwan {tyyen,yylee}@nlg.csie.ntu.edu.tw, orina1123@gmail.com, hhhuang@nccu.edu.tw, hhchen@ntu.edu.tw Abstract Sense embedding models handle polysemy by giving each distinct meaning of a word form a separate representation. They are considered improvements over word models, and their effectiveness is usually judged with benchmarks such as semantic similarity datasets. However, most of these datasets are not designed for evaluating sense embeddings. In this research, we show that there are at least six concerns about evaluating sense embeddings with existing benchmark datasets, including the large proportions of single-sense words and the unexpected inferior performance of several multi-sense models to their single-sense counterparts. These observations call into serious question whether evaluations based on these datasets can reflect the sense model’s ability to capture different meanings. To address the issues, we propose the Multi-Sense Dataset (MSD-1030), which contains a high ratio of multi-sense word pairs. A series of analyses and experiments show that MSD-1030 serves as a more reliable benchmark for sense embeddings. The dataset is available at http://nlg.csie.ntu.edu.tw/nlpresource/MSD-1030/. Keywords: semantics, evaluation methodologies, crowdsourcing 1. Introduction Word embeddings, or distributed word representations, have attracted much attention in recent years. Previous studies show that word embedding models are capable of learning semantic and syntactic information from a large unannotated corpus (Mikolov et al., 2013; Pennington et al., 2014). However, one essential issue in word embeddings is that each word form is represented by only one vector. That is, multiple senses of a word form are indistinguishable, which is problematic for applications involving word ambiguity such as word sense disambiguation (WSD) and semantic relation identification. Sense embedding models, in which each sense of a word form is represented by its own vector (Reisinger and Mooney, 2010; Huang et al., 2012; Jauhar et al., 2015; Bartunov et al., 2016; Lee and Chen, 2017; Lee et al., 2018), have been proposed to address the polysemy issue mentioned above. Camacho-Collados and Pilehvar (2018) provide an extensive review of previous studies in sense embeddings. More recently, pre-trained contextualized word representations such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) handle polysemy by assigning a vector representation conditioned on the specific context to every word in a sentence. Different from those approaches, sense embeddings can be grounded in an ontology such as WordNet (Miller, 1995) or BabelNet (Navigli and Ponzetto, 2012), which can support operations such as queries and making inference over the ontology or any connected knowledge base. On the other hand, sense embeddings can be trained with smaller amounts of data and further enhanced by external ontologies, making them extremely useful in …

Content truncated to 500 words