Sense-annotating a Lexical Substitution Data Set with Ubyline

Sense-annotating a Lexical Substitution Data Set with Ubyline Tristan Miller, Mohamed Khemakhem, Richard Eckart de Castilho, and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universit¨at Darmstadt https://www.ukp.tu-darmstadt.de/ Abstract We describe the construction of GLASS, a newly sense-annotated version of the German lexical substitution data set used at the GERMEVAL 2015: LEXSUB shared task. Using the two annotation layers, we conduct the first known empirical study of the relationship between manually applied word senses and lexical substitutions. We find that synonymy and hypernymy/hyponymy are the only semantic relations directly linking targets to their substitutes, and that substitutes in the target’s hypernymy/hyponymy taxonomy closely align with the synonyms of a single GermaNet synset. Despite this, these substitutes account for a minority of those provided by the annotators. The results of our analysis accord with those of a previous study on English-language data (albeit with automatically induced word senses), leading us to suspect that the sense–substitution relations we discovered may be of a universal nature. We also tentatively conclude that relatively cheap lexical substitution annotations can be used as a knowledge source for automatic WSD. Also introduced in this paper is Ubyline, the web application used to produce the sense annotations. Ubyline presents an intuitive user interface optimized for annotating lexical sample data, and is readily adaptable to sense inventories other than GermaNet. Keywords: lexical substitutions, word senses, word sense annotation 1 Introduction Word sense disambiguation (WSD)—the task of determin- ing a word’s meaning in context—is one of the oldest open research problems in computational linguistics. WSD sys- tems are commonly evaluated by having humans mark up the words in a text with their contextually appropriate meanings, as enumerated by a dictionary or other lexical-semantic re- source, and then comparing these annotations against those supplied by the systems. Such “in vitro” evaluations are popular because they are straightforward to conduct, though they have the disadvantage of requiring considerable effort to produce the manually annotated gold standard data. Sense- annotated data sets remain rare, particularly for languages other than English. A more recent and increasingly popular evaluation method, which has the advantage of not requiring all human and machine annotators to use the same sense inventory, is lex- ical substitution. Here the lexical annotations which are applied and compared are not sense labels, but rather lists of plausible synonyms. It has been argued that, since the identification and ranking of these substitutions depends on a proper understanding of the word’s meaning in context, accuracy in this “in vivo” task is an indirect measure of WSD performance (McCarthy, 2002). The contributions of the present work are twofold. First, to ease the considerable technical and ergonomic burdens of creating sense-annotated data, we provide Ubyline,1 an Apache-licensed, web-based sense annotation tool whose user interface is optimized for lexical sample data. (Lexical sample data sets feature documents or short texts in which annotations are applied to all instances of a fixed set of lem- mas.) Ubyline supports a wide range of sense inventories …

Content truncated to 500 words