UPB at SemEval-2020 Task 6: Pretrained Language Models for Definition Extraction

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 737–745 Barcelona, Spain (Online), December 12, 2020. 737 UPB at SemEval-2020 Task 6: Pretrained Language Models for Definition Extraction Andrei-Marius Avram1,2, Dumitru-Clementin Cercel1, Costin-Gabriel Chiru1 University Politehnica of Bucharest, Faculty of Automatic Control and Computers1 Research Institute for Artificial Intelligence, Romanian Academy2 avram.andreimarius@gmail.com, {clementin.cercel, costin.chiru}@cs.pub.ro Abstract This work presents our contribution in the context of the 6th task of SemEval-2020: Extracting Definitions from Free Text in Textbooks (DeftEval). This competition consists of three subtasks with different levels of granularity: (1) classification of sentences as definitional or non-definitional, (2) labeling of definitional sentences, and (3) relation classification. We use various pretrained language models (i.e., BERT, XLNet, RoBERTa, SciBERT, and ALBERT) to solve each of the three subtasks of the competition. Specifically, for each language model variant, we experiment by both freezing its weights and fine-tuning them. We also explore a multi-task architecture that was trained to jointly predict the outputs for the second and the third subtasks. Our best performing model evaluated on the DeftEval dataset obtains the 32nd place for the first subtask and the 37th place for the second subtask. The code is available for further research at: https: //github.com/avramandrei/DeftEval . 1 Introduction Definition extraction from text is a challenging research task, addressed by numerous researchers in the area of natural language processing (NLP). Factual question-answering systems are possible applications that can benefit from the results of this task (Zhang and Jiang, 2009). As a response to this challenge, Spala et al. (2019) introduced the Definition Extraction from Texts (DEFT) corpus, a human-annotated English dataset that contains multi-domain (e.g., biology, sociology, physics) term-definition pairs from two types of documents, free (i.e., Textbook) and semi-structured text (i.e., Contracts), as opposed to the domain-specific WCL dataset (Navigli and Velardi, 2010). In addition, a shared task was proposed by SemEval-2020 which was aimed at evaluating the performance of each participant system on three subtasks defined for the DEFT corpus. The three subtasks are the following: Subtask 1: Given a labeled dataset of sentences, this subtask is to build a classifier capable of distinguishing between a sentence containing both a definition and the defined term or not. We provide two examples of sentences that contain a definition, from Biology and Physics domains, respectively: • The metabolome is the complete set of metabolites that are related to the genetic makeup of an organism. • Polarization is the separation of charges in an object that remains neutral. Subtask 2: Given a dataset of tokenized sentences, we aim to label each token with one of the following classes: Term, Alias-Term, Referential-Term, Definition, Referential-Definition, or Qualifier. The meaning of each token can be found in the corpus description paper (Spala et al., 2019). Subtask 3: Given a dataset of tokenized sentences and the tag id of each token, our goal is to predict, for each token, the type of relationship it had with another token, and also the tag id of the token it had a relation …

Content truncated to 500 words