Out-of-domain FrameNet Semantic Role Labeling

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 471–482, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Out-of-domain FrameNet Semantic Role Labeling Silvana Hartmann§†, Ilia Kuznetsov†, Teresa Martin§†, Iryna Gurevych§† §Research Training Group AIPHES †Ubiquitous Knowledge Processing (UKP) Lab Department of Computer Science, Technische Universit¨at Darmstadt http://www.ukp.tu-darmstadt.de Abstract Domain dependence of NLP systems is one of the major obstacles to their application in large-scale text analysis, also restrict- ing the applicability of FrameNet semantic role labeling (SRL) systems. Yet, current FrameNet SRL systems are still only eval- uated on a single in-domain test set. For the first time, we study the domain depen- dence of FrameNet SRL on a wide range of benchmark sets. We create a novel test set for FrameNet SRL based on user-generated web text and find that the major bottleneck for out-of-domain FrameNet SRL is the frame identification step. To address this problem, we develop a simple, yet efficient system based on distributed word repre- sentations. Our system closely approaches the state-of-the-art in-domain while outper- forming the best available frame identifica- tion system out-of-domain. We publish our system and test data for research purposes.1 1 Introduction Domain dependence is a major problem for super- vised NLP tasks such as FrameNet semantic role labeling (SRL): systems generally exhibit a strong performance drop when applied to test data from a different distribution than the training data. This prohibits their large-scale use in language technol- ogy applications. The same problems are expected for FrameNet SRL, but due to a lack of datasets, state-of-the- art FrameNet SRL is only evaluated on a single in-domain test set, see e.g. Das et al. (2014) and FitzGerald et al. (2015). In this work, we present the first comprehensive study of the domain dependence of FrameNet SRL 1www.ukp.tu-darmstadt.de/ood-fn-srl on a range of benchmark datasets. This is crucial as the demand for semantic textual analysis of large- scale web data keeps growing. Based on FrameNet (Fillmore et al., 2003), FrameNet SRL extracts frame-semantic structures on the sentence level that describe a specific situation centered around a semantic predicate, often a verb, and its participants, typically syntactic arguments or adjuncts of the predicate. The predicate is assigned a frame label, essentially a word sense label, that defines the situation and determines the semantic roles of the participants. The following sentence from FrameNet provides an example of the Grinding frame and its roles: [The mill]Grinding cause grindsGrinding [the malt]Patient [to grist]Result. FrameNet SRL consists of two steps, frame iden- tification (frameId), assigning a frame to the current predicate, and role labeling (roleId), identifying the participants and assigning them role labels licensed by the frame. The frameId step reduces the hun- dreds of role labels in FrameNet to a manageable set of up to 30 roles. Thus, FrameNet SRL dif- fers from PropBank SRL (Carreras and M`arquez, 2005), that only uses a small set of 26 syntactically motivated role labels and puts less weight …

Content truncated to 500 words