SemEval-2020 Task 6: Definition Extraction from Free Text with the DEFT Corpus

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 336–345 Barcelona, Spain (Online), December 12, 2020. 336 SemEval-2020 Task 6: Definition extraction from free text with the DEFT corpus Sasha Spala1, Nicholas A Miller1, Franck Dernoncourt2, Carl Dockhorn1 1Adobe Inc., 2Adobe Research 345 Park Ave San Jose, CA 95110-2704 {sspala,nimiller,dernonco,cdockhorn}@adobe.com Abstract Research on definition extraction has been conducted for well over a decade, largely with sig- nificant constraints on the type of definitions considered. In this work, we present DeftEval, a SemEval shared task in which participants must extract definitions from free text using a term- definition pair corpus that reflects the complex reality of definitions in natural language. Defi- nitions and glosses in free text often appear without explicit indicators, across sentences bound- aries, or in an otherwise complex linguistic manner. DeftEval involved 3 distinct subtasks: 1) Sentence classification, 2) sequence labeling, and 3) relation extraction. 1 Introduction Definition extraction as a complex, real-world task is currently an emerging field of study. Traditional definition extraction approaches mostly rely on simple, syntactically straight-forward examples with relatively little variance in vocabulary. Corpora, including the WCL (Navigli et al., 2010) and ukWaC (Ferraresi et al., 2008), typically consist of “definition sentences” which follow a standard X is a (type) Y or X, such as Y syntactic structure. Many also contain definitors (Navigli and Velardi, 2010), such as “means”, “is”, or “is defined by”. We provide a complete review of the existing state of definition extraction in Spala et al. (2019). 2 Data The DeftEval task provided data from the DEFT corpus for training, development and testing. Introduced in Spala et al (2019), the DEFT corpus is currently the largest and most comprehensive corpus explicitly for definition extraction. In addition to the typical definition-type sentences discussed in the introduction, the corpus also contains sentence and “sentence windows” of wide variance in syntactic and semantic construction. Sentences for the corpus were retrieved from the open source textbook website cnx.org, as well as from various 2017 SEC contract filings from the publicly available US Securities and Exchange Commission EDGAR (SEC) database. For the DeftEval shared task, participants were only required to use sentences extracted from textbooks. EDGAR annotations were provided on request, but not required for participation in the shared task. The DEFT corpus provides sentence “context windows” for textbook data from cnx.org to narrow the search space for possible definitions. A context window is defined as a set of three sentences: one sentence before a sentence containing a potential term (e.g. a bold word), the main sentence with the potential term, and one sentence after. Though it is certainly possible that some definitions may occur further away from the bolded mention of the term, for the purposes of the shared task, definitions that occur outside of the three-sentence window around a potential term are out of scope. The corpus contains 9 different token labels (see Table1), and 5 different types of relations (see Ta- ble2). This expanded annotation schema allows for a wide …

Content truncated to 500 words