Defx at SemEval-2020 Task 6: Joint Extraction of Concepts and Relations for Definition Extraction

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 704–709 Barcelona, Spain (Online), December 12, 2020. 704 defx at SemEval-2020 Task 6: Joint Extraction of Concepts and Relations for Definition Extraction Marc H¨ubner, Christoph Alt, Robert Schwarzenberg, Leonhard Hennig German Research Center for Artificial Intelligence (DFKI), Berlin, Germany {firstname.lastname@dfki.de} Abstract Definition Extraction systems are a valuable knowledge source for both humans and algorithms. In this paper we describe our submissions to the DeftEval shared task (SemEval-2020 Task 6), which is evaluated on an English textbook corpus. We provide a detailed explanation of our system for the joint extraction of definition concepts and the relations among them. Furthermore we provide an ablation study of our model variations and describe the results of an error analysis. 1 Introduction Definition extraction (DE) is a subfield of information extraction that deals with the automated extraction of terms and their descriptions from natural language text. We belief that for humans definitions are one of the most important sources of knowledge to clarify unknown words in a new language or domain. Thus, an important application is the automated construction of a glossary, which is a laborious task if done manually. Another important application is the use of definitions as background knowledge for machine learning algorithms. Recent results have disclosed difficulties of state of the art approaches when confronted with factual knowledge (Logan et al., 2019), and have lead to a surge of research in this area (Logeswaran et al., 2019; Zhang et al., 2019; Peters et al., 2019). The advantages of definitions as a knowledge source are two-fold: Firstly, definitions are easier to extract and annotate than other forms of knowledge sources, e.g. Knowledge Graphs (KG). And secondly, definitions can be processed using the same text-based algorithms whereas the use of a KG requires other forms of algorithms, e.g., Graph Embeddings. While research using definitions exists, it is limited by the scale of available annotations and restricts large scale applications such as in self-supervised learning setups. This paper describes our submission to the DeftEval challenge Subtask 2. Our approach is based on a multi-task learning strategy, that jointly extracts concepts and relations between them, and achieves a score of 49.68 macro F1, which ranked our best system in place 27 of 51 participants. The major challenge turned out to be the strong label imbalance that made an evaluation using the official macro-averaged F1 score difficult. The source code of our approach is publicly available.1 2 DeftEval Challenge The DeftEval challenge2 (Spala et al., 2020) includes three subtasks: (1) classification of definition sentences, (2) sequence labeling of definition concepts and (3) relation extraction between concepts. The DEFT corpus (Spala et al., 2019) is used for evaluation, which consists of English textbooks scraped from an online-learning website.3 Compared to the previous manually annotated datasets WCL (Navigli et al., 2010) and W00 (Jin et al., 2013), the DEFT corpus is significantly larger and the examples are more diverse than simple is-a patterns. In contrast to WCL and W00 …

Content truncated to 500 words