SemEval-2019 Task 2: Unsupervised Lexical Frame Induction

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 16–30 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 16 SemEval-2019 Task 2: Unsupervised Lexical Frame Induction Behrang QasemiZadeh SFB991, Germany zadeh@phil.hhu.de Miriam R. L. Petruck ICSI, US miriamp@icsi.berkeley.edu Regina Stodden HHUD, Germany stodden@phil.hhu.de Laura Kallmeyer HHUD, SFB991, Germany kallmeyer@phil.hhu.de Marie Candito Paris Diderot University - CNRS, France marie.candito@linguist.univ-paris-diderot.fr Abstract This paper presents Unsupervised Lexical Frame Induction, Task 2 of the International Workshop on Semantic Evaluation in 2019. Given a set of prespecified syntactic forms in context, the task requires that verbs and their arguments be clustered to resemble semantic frame structures. Results are useful in iden- tifying polysemous words, i.e., those whose frame structures are not easily distinguished, as well as discerning semantic relations of the arguments. Evaluation of unsupervised frame induction methods fell into two tracks: Task A) Verb Clustering based on FrameNet 1.7; and B) Argument Clustering, with B.1) based on FrameNet’s core frame elements, and B.2) on VerbNet 3.2 semantic roles. The shared task attracted nine teams, of whom three re- ported promising results. This paper describes the task and its data, reports on methods and resources that these systems used, and offers a comparison to human annotation. 1 Introduction SemEval 2019 Task 2 focused on the unsupervised semantic labeling of a set of prespecified (seman- tically) unlabeled structures (Figure 1). Unsuper- vised learning methods analyze these structures (Figure 1a) to augment them with semantic labels (Figure 1b). The shape of the manually labeled in- put frames is constrained to an acyclic connected tree of lexical items (words and multi-word units) of maximum depth 1, where just one root gov- erns several arguments. The task used Berkeley FrameNet (FN) (Ruppenhofer et al., 2016) and Q. Zadeh and Petruck (2019), guidelines for this task, to determine the arguments and label them with semantic information. We compared the proposed system results for unsupervised semantic tagging with that of human annotated (or, gold-standard) data in three differ- ent subtasks (Figure 2). To evaluate the systems, we computed distributional similarities between Exxon Mobil sell skyscraper company nsubj dobj nmod:to (a) Input: subcategorization frames. Exxon Mobil sell COMMERCE SELL skyscraper company nsubj Seller Agent dobj Goods Theme nmod:to Buyer Recipient (b) Output: Semantic Frame Tagging using labels learned by Unsupervised methods. Figure 1: Given semantically unlabeled structures (1a), annotate the input with semantic information learned via unsupervised methods (1b). their generated unsupervised labeled data and hu- man annotated reference data. For computing sim- ilarities we used general purpose numeral methods of text clustering, in particular BCUBED F-SCORE (Bagga and Baldwin, 1998) as the single figure of merit to rank the systems. The most important result of the shared task is the creation of a benchmark for a future complex task. This benchmark includes a moderately sized, manually annotated set of frames, where only the verbs of each were included, along with their core frame elements (which uniquely define a frame as Ruppenhofer et al. describe). To complement …

Content truncated to 500 words