SemEval-2016 Task 10: Detecting Minimal Semantic Units and their Meanings (DiMSUM)

Proceedings of SemEval-2016, pages 546–559, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval-2016 Task 10: Detecting Minimal Semantic Units and their Meanings (DiMSUM) Nathan Schneider School of Informatics University of Edinburgh Edinburgh, UK Dirk Hovy Anders Johannsen Center for Language Technology University of Copenhagen Copenhagen, Denmark nschneid@inf.ed.ac.uk, krx628@hum.ku.dk, anders@johannsen.com, marine@cs.umd.edu Marine Carpuat Computer Science Dept. University of Maryland College Park, Maryland, USA Abstract This task combines the labeling of multiword expressions and supersenses (coarse-grained classes) in an explicit, yet broad-coverage paradigm for lexical semantics. Nine sys- tems participated; the best scored 57.7% F1 in a multi-domain evaluation setting, indicating that the task remains largely unresolved. An error analysis reveals that a large number of instances in the data set are either hard cases, which no systems get right, or easy cases, which all systems correctly solve. 1 Introduction Grammatical analysis tasks, e.g., part-of-speech tag- ging, are rather successful applications of natural language processing (NLP). They are comprehen- sive, i.e., they operate under the assumption that all grammatically-relevant parts of a sentence will be analyzed: We do not expect a POS tagger to only know a subset of the tags in the language. Most POS tags accommodate unseen words and adapt readily to new text genres. Together, these factors indicate a representation which achieves broad coverage. Explicit analysis of lexical semantics, by contrast, has been more difficult to scale to broad coverage owing to limited comprehensiveness and extensibil- ity. The dominant paradigm of fine-grained word sense disambiguation, WordNet (Fellbaum, 1998), is difficult to annotate in corpora, results in consider- able data sparseness, and does not readily generalize to out-of-vocabulary words. While the main corpus with WordNet senses, SemCor (Miller et al., 1993), does reflect several text genres, it is hard to expand SemCor-style annotations to new genres, such as so- cial web text or transcribed speech. This severely limits the applicability of SemCor-based NLP tools and restricts opportunities for linguistic studies of lexical semantics in corpora. To address this limitation, in the DiMSUM 2016 shared task,1 we challenged participants to analyze the lexical semantics of English sentences with a tagset integrating multiword expressions and noun and verb supersenses (following Schneider and Smith, 2015), on multiple nontraditional genres of text. By moving away from fine-grained sense in- ventories and lexicalized, language-specific2 anno- tation, we take a step in the direction of broad- coverage, coarse-grained lexical semantic analysis. We believe this departure from the classical lexical semantics paradigm will ultimately prove fruitful for a variety of NLP applications in a variety of genres. The integrated lexical semantic representation (§2, §3) has been annotated in an extensive bench- mark data set comprising several nontraditional do- mains (§4). Objective, controlled evaluation proce- dures (§5) facilitate a comparison of the 9 systems submitted as part of the official task (§6). While the systems range in performance, all are below 60% in our composite evaluation, suggesting that further work is needed to make progress on this difficult task. 2 Background Multiword expressions. Most …

Content truncated to 500 words