Cardiff University at SemEval-2020 Task 6: Fine-tuning BERT for Domain-Specific Definition Classification

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 361–366 Barcelona, Spain (Online), December 12, 2020. 361 Cardiff University at SemEval-2020 Task 6: Fine-tuning BERT for Domain-Specific Definition Classification Shelan S. Jeawak, Luis Espinosa-Anke and Steven Schockaert School of Computer Science and Informatics Cardiff University, UK {jeawakss,espinosa-ankel,schockaerts1}@cardiff.ac.uk Abstract We describe the system submitted to SemEval-2020 Task 6, Subtask 1. The aim of this subtask is to predict whether a given sentence contains a definition or not. Unsurprisingly, we found that strong results can be achieved by fine-tuning a pre-trained BERT language model. In this paper, we analyze the performance of this strategy. Among others, we show that results can be improved by using a two-step fine-tuning process, in which the BERT model is first fine-tuned on the full training set, and then further specialized towards a target domain. 1 Introduction Definitions are central to the way in which humans convey knowledge about the meaning of concepts. Accordingly, a large number of general and domain-specific dictionaries have been created. As new concepts emerge, and the meaning of existing concepts shifts, these dictionaries need to be updated. This continual process is traditionally carried out by linguists or domain experts, meaning that dictionaries are never fully up-to-date. In rapidly evolving scientific domains, among others, this is a clear limitation. An appealing alternative is to automatically identify and extract definitions expressed in free text. This task of extracting term-definition pairs from text corpora is known as Definition Extraction (DE). Early attempts to solve this task relied on rule-based methods (Klavans and Muresan, 2001; Cui et al., 2005). However, such methods are typically only able to detect explicit, direct and structured definitions, which usually contain definitor verb phrases such as means, is, is defined as. Later, a large number of supervised and semi supervised machine learning models for DE have been proposed (Westerhout, 2009; Reiplinger et al., 2012; Jin et al., 2013). While being able to identify a wider range of definitions, these approaches cannot be adapted to new domains efficiently, as they crucially rely on carefully designed features, which might not be available, or be less effective, in the new domain. More recently, the focus has shifted to neural network based models (Espinosa-Anke and Schockaert, 2018; Veyseh et al., 2019). The method we analyze in this paper is based on fine-tuning a pre-trained BERT language model (Devlin et al., 2018). This strategy has recently proven successful across a wide range of Natural Language Processing (NLP) tasks. In particular, we focus on SemEval-2020 Task 6: DeftEval: Extracting term-definition pairs in free text (Spala et al., 2020). We participated in Subtask 1: Sentence Classification. This task required participants to predict whether a given sentence contains a definition. The associated dataset contains documents from seven different domains, including biology, history and economics. In our analysis, we focus on comparing two different strategies for fine-tuning the pre-trained BERT model: 1. fine-tuning a single BERT model based on all the available training data; 2. fine-tuning a separate BERT …

Content truncated to 500 words