Is Language Modeling Enough? Evaluating Effective Embedding Combinations

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4739–4748 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4739 Is Language Modeling Enough? Evaluating Effective Embedding Combinations Rudolf Schneider∗, Tom Oberhauser∗, Paul Grundmann∗, Felix A. Gers∗ Alexander L¨oser∗, Steffen Staab† ∗Beuth University of Applied Sciences Berlin, Luxemburger Str. 10, 13353 Berlin {ruschneider, toberhauser, pgrundmann, gers, aloeser}@beuth-hochschule.de †University Stuttgart, Universitaetsstrasse 32, 70569 Stuttgart, Germany †University of Southampton, University Road, Southampton, SO17 1BJ, United Kingdom Steffen.Staab@ipvs.uni-stuttgart.de Abstract Universal embeddings, such as BERT or ELMo, are useful for a broad set of natural language processing tasks like text classification or sentiment analysis. Moreover, specialized embeddings also exist for tasks like topic modeling or named entity disambiguation. We study if we can complement these universal embeddings with specialized embeddings. We conduct an in-depth evaluation of nine well known natural language understanding tasks with SentEval. Also, we extend SentEval with two additional tasks to the medical domain. We present PubMedSection, a novel topic classification dataset focussed on the biomedical domain. Our comprehensive analysis covers 11 tasks and combinations of six embeddings. We report that combined embeddings outperform state of the art universal embeddings without any embedding fine-tuning. We observe that adding topic model based embeddings helps for most tasks and that differing pre-training tasks encode complementary features. Moreover, we present new state of the art results on the MPQA and SUBJ tasks in SentEval. Keywords: representation learning, meta embedding, evaluation, neural language representation models 1. Introduction Universal embeddings, such as BERT (Devlin et al., 2019) or ELMo (Peters et al., 2018), are an effective text represen- tation (Nguyen et al., 2016; Conneau and Kiela, 2018). Of- ten, they are trained on hundreds of millions of documents with an language modeling objective and contain millions to even billions of parameters. These pre-trained vec- tors lead to significant increases in performance in various downstream natural language processing tasks (Mikolov et al., 2013a; Joulin et al., 2017; Akbik et al., 2018; Peters et al., 2018; Radford et al., 2018). Contrary to universal em- beddings, specialized embeddings for tasks like entity link- ing (Pappu et al., 2017; Gillick et al., 2019) or paragraph classification (Arnold et al., 2019) exist. Often, special- ized embeddings are trained with objectives and training datasets different from universal embeddings. This circum- stance raises the question if universal embeddings capture all useful features for downstream tasks or if specialized embeddings may provide complementary features. Example: Clinical Decision Support Systems. Medi- cal literature databases, such as PubMed1 or UpToDate2, help doctors answer their questions. These systems bene- fit from methods that enrich texts with semantic concepts, like entity recognition, sentence classification, topic clas- sification, or relation extraction (Demner-Fushman et al., 2009; Berner, 2007). Medical language is highly special- ized and often ambiguous in clinical documents (Leaman et al., 2015). Documents, such as medical research pa- pers, doctors’ letters or clinical notes, are heterogeneous in terms of structure, vocabulary, or grammatical correct- ness (Starlinger et al., 2017). We …

Content truncated to 500 words