Transforming Term Extraction: Transformer-Based Approaches to Multilingual Term Extraction Across Domains

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3607–3620 August 1–6, 2021. ©2021 Association for Computational Linguistics 3607 Transforming Term Extraction: Transformer-Based Approaches to Multilingual Term Extraction Across Domains Christian Lang∗, Lennart Wachowiak*, Barbara Heinisch, Dagmar Gromann University of Vienna, Center for Translation Studies [christian.lang,lennart.wachowiak]@univie.ac.at Abstract Automated Term Extraction (ATE), even though well-investigated, continues to be a challenging task. Approaches conventionally extract terms on corpus or document level and the benefits of neural models still remain un- derexplored with very few exceptions. We in- troduce three transformer-based term extrac- tion models operating on sentence level: a lan- guage model for token classification, one for sequence classification, and an innovative use of Neural Machine Translation (NMT), which learns to reduce sentences to terms. All three models are trained and tested on the dataset of the ATE challenge TermEval 2020 in English, French, and Dutch across four specialized do- mains. The two best performing approaches are also evaluated on the ACL RD-TEC 2.0 dataset. Our models outperform previous base- lines, one of which is BERT-based, by a sub- stantial margin, with the token-classifier lan- guage model performing best. 1 Introduction Automated Term Extraction (ATE) aims at extract- ing terms, i.e., single- or multi-word sequences, from domain-specific text. ATE plays a role in many NLP tasks, such as information extraction, knowledge graph learning, and text summariza- tion. In a corpus-level setting, methods range from frequency-based to utilizing Wikipedia links, where no single method has been found to perform consistently best across domains in English (As- trakhantsev, 2018). In document-level ATE, Key- ConceptRelatedness (Astrakhantsev, 2014), which relies on keyphrase extraction and semantic related- ness, outperforms other methods (ˇSajatovi´c et al., 2019). The use of neural networks in these methods is mostly limited to generating embeddings. A first use of BERT-based language models is documented by Hazem et al. (2020), the winning ∗* Equal contributions system of the recent ATE challenge TermEval 2020 (Rigouts Terryn et al., 2020) and the baseline for the proposed approaches. Inspired by this first success of transformer-based models, we compare two variations of the multilingual pretrained lan- guage model XLM-RoBERTa (XLM-R) (Conneau et al., 2020) with an innovative use of the multi- lingual pretrained NMT model mBART (Liu et al., 2020) on the Annotated Corpora for Term Extrac- tion Research (ACTER) dataset (Rigouts Terryn et al., 2019) utilized in TermEval 2020 as well as on the ACL RD-TEC 2.0 dataset (QasemiZadeh and Schumann, 2016). Since masked language and NMT models take sentences as input, the proposed ATE methods operate on sentence level. In spite of this reduced context of sentence input rather than documents or corpora, the models achieve F1 scores of up to 69.8% on ACTER, strongly outper- forming the previous baseline of 48.1% . An XLM-R-based sequence classifier relies on positive (term) and negative (non-term) samples, which are generated based on all n-grams up to a length of six of a given sentence. A second XLM- R-based token classifier decides for each word in a sequence …

Content truncated to 500 words