FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8382–8395 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8382 FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations Lukas Lange1,2,3 Heike Adel1 1 Bosch Center for Artificial Intelligence, Renningen, Germany 2 Spoken Language Systems (LSV), Saarland University, Saarbr¨ucken, Germany 3 Saarbr¨ucken Graduate School of Computer Science, Saarbr¨ucken, Germany {Lukas.Lange,Heike.Adel,Jannik.Stroetgen}@de.bosch.com dietrich.klakow@lsv.uni-saarland.de Jannik Str¨otgen1 Dietrich Klakow2 Abstract Combining several embeddings typically im- proves performance in downstream tasks as different embeddings encode different infor- mation. It has been shown that even models us- ing embeddings from transformers still bene- fit from the inclusion of standard word embed- dings. However, the combination of embed- dings of different types and dimensions is chal- lenging. As an alternative to attention-based meta-embeddings, we propose feature-based adversarial meta-embeddings (FAME) with an attention function that is guided by features reflecting word-specific properties, such as shape and frequency, and show that this is ben- eficial to handle subword-based embeddings. In addition, FAME uses adversarial training to optimize the mappings of differently-sized embeddings to the same space. We demon- strate that FAME works effectively across lan- guages and domains for sequence labeling and sentence classification, in particular in low- resource settings. FAME sets the new state of the art for POS tagging in 27 languages, var- ious NER settings and question classification in different domains. 1 Introduction Recent work on word embeddings and pre-trained language models has shown the large impact of language representations on natural language pro- cessing (NLP) models across tasks and domains (Devlin et al., 2019; Beltagy et al., 2019; Conneau et al., 2020). Nowadays, a large number of differ- ent embedding models are available with different characteristics, such as different input granularities (word-based (e.g., Mikolov et al., 2013; Pennington et al., 2014) vs. subword-based (e.g., Heinzerling and Strube, 2018; Devlin et al., 2019) vs. character- based (e.g., Lample et al., 2016; Ma and Hovy, 2016; Peters et al., 2018)), or different data used for pre-training (general-world vs. specific domain). Since those characteristics directly influence when embeddings are most effective, combinations of different embedding models are likely to be benefi- cial (Tsuboi, 2014; Kiela et al., 2018; Lange et al., 2019b), even when using already powerful large- scale pre-trained language models (Akbik et al., 2018; Yu et al., 2020). Word-based embeddings, for instance, are strong in modeling frequent words while character-based embeddings can model out- of-vocabulary words. Similarly, domain-specific embeddings can capture in-domain words that do not appear in general domains like news text. Different word representations can be combined using so-called meta-embeddings. There are sev- eral methods available, ranging from concatena- tion (e.g., Yin and Sch¨utze, 2016), over averag- ing (e.g., Coates and Bollegala, 2018) to attention- based meta-embeddings (Kiela et al., 2018). How- ever, they all come with shortcomings: Concatena- tion leads to high-dimensional input vectors and, as a result, requires additional parameters in the first layer of the neural network. Averaging simply merges all information into one vector, not allowing the network …

Content truncated to 500 words