CLaC-BP at SemEval-2021 Task 8: SciBERT Plus Rules for MeasEval

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 410–415 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 410 CLaC-BP at SemEval-2021 Task 8: SciBERT Plus Rules for MeasEval Benjamin Th´erien, Parsa Bagherzadeh, and Sabine Bergler CLaC Labs, Concordia University Montreal, Canada {b therie, p bagher, bergler} @cse.concordia.ca Abstract This paper explains the design of a heteroge- neous system that ranked eighth in competi- tion in SemEval2021 Task 8. We analyze abla- tion experiments and demonstrate how the sys- tem components, namely tokenizer, unit iden- tifier, modifier classifier, and language model, affect the overall score. We compare our re- sults to similar experiments from the literature and introduce a grouping algorithm developed in the post-evaluation phase that increased our system’s overall score, hypothetically elevat- ing our competition rank from eight to six. 1 Introduction The MeasEval 2021 shared task involves identi- fying related groups of the four annotation types seen in Table 1, identifying relations linking the an- notations of each group, and providing additional information (units and modifiers) for quantities (see (Harper et al., 2021)). Our system learns to classify tokens, groups tokens of the same class into spans and further groups related spans using a distance- based heuristic, providing a baseline for systems that attempt to learn these groupings. 2 Data The data comprises excerpts from scientific articles of different disciplines: Astronomy, Engineering, Medicine, Agriculture, Biology, Chemistry, Earth Science, Computer Science, and Mathematics. The limited data (see Table 1), the wide range of top- ics involved, and the idiosyncrasies of each scien- tific field’s vocabulary make the task difficult. In fact, the task organizers report a high degree of inter-annotator disagreement for certain annotation types. Krippendorff’s alpha (Hayes and Krippen- dorff, 2007) for each annotation type is also shown in Table 1, underlining that annotating Measured Properties, Measured Entities, and Qualifiers poses a substantial degree of ambiguity, even to humans. Annotation Frequency α Quantity (QA) 1164 .94 Unit (U) .86 MeasuredEntity (ME) 1148 .54 MeasuredProperty (MP) 742 .64 Qualifier (QL) 276 .33 Table 1: Frequency of training data annotations and their Krippendorff’s Alpha The training data contains a total of 298 excerpts containing 1164 different quantities. We refer to groupings of annotations as complete data-points (see Figure 1). All complete data-points contain one quantity and at most one annotation span of each other type. A summary of complete data-point frequencies is provided in Table 2. Annotations present Frequency QA 11 ME,QA 361 ME,MP,QA 512 ME,QL,QA 50 ME,MP,QL,QA 225 MP,QA 4 MP,QL,QA 1 Table 2: Frequency of different complete data-points in the training data 3 System Our system is a pipeline of different machine learn- ing and rule-based modules. We use SpaCy1 for sentence splitting and tokenization and we fine- tune a SciBERT model2 to classify each token into one of Quantity (QA), Measured Entity (ME), Mea- sured Property (MP), Qualifier (QL), or Other (O). 1https://spacy.io/ 2SciBERT uses the same model architecture and pretrain- ing objective as BERT (Devlin et al., 2019), but it …

Content truncated to 500 words