Attribute Injection for Pretrained Language Models: A New Benchmark and an Efficient Method

Proceedings of the 29th International Conference on Computational Linguistics, pages 1051–1064 October 12–17, 2022. 1051 Attribute Injection for Pretrained Language Models: A New Benchmark and An Efficient Method Reinald Kim Amplayo∗ University of Edinburgh reinald.kim@ed.ac.uk Kang Min Yoo Sang-Woo Lee NAVER AI Lab, NAVER Clova AI {kangmin.yoo,sang.woo.lee}@navercorp.com Abstract Metadata attributes (e.g., user and product IDs from reviews) can be incorporated as addi- tional inputs to neural-based NLP models, by expanding the architecture of the models to im- prove performance. However, recent models rely on pretrained language models (PLMs), in which previously used techniques for at- tribute injection are either nontrivial or cost- ineffective. In this paper, we introduce a benchmark for evaluating attribute injection models, which comprises eight datasets across a diverse range of tasks and domains and six synthetically sparsified ones. We also propose a lightweight and memory-efficient method to inject attributes into PLMs. We extend adapters, i.e. tiny plug-in feed-forward mod- ules, to include attributes both independently of or jointly with the text. We use approxi- mation techniques to parameterize the model efficiently for domains with large attribute vo- cabularies, and training mechanisms to han- dle multi-labeled and sparse attributes. Exten- sive experiments and analyses show that our method outperforms previous attribute injec- tion methods and achieves state-of-the-art per- formance on all datasets. 1 Introduction Neural-based NLP models are powered by large- scale textual datasets, which are mostly crawled from the web (Denoyer and Gallinari, 2006; Sand- haus, 2008; Zhu et al., 2015; Ni et al., 2019; Raffel et al., 2020). Web texts usually are attached with metadata, i.e. attributes that describe the texts. For example, product reviews have user and prod- uct IDs, as well as their ratings, while research papers on arXiv have author lists and research ar- eas as metadata attributes (see Figure 1). While most of the recent models disregard them and fo- cus more on ungrounded language understanding ∗ Work done while Reinald was at NAVER AI Lab. He is now at Google Research. Yelp Review Text: My boyfriend’s fav. place and the stein of beers are priced pretty good. Game nights get super packed so go early to save a seat. Kitchen closes at midnight which is too early when your buzz kicks in around 1am. Attributes: – User: n6LeAoIuDR3NfIBEsmL_zg – Product: 7TMf1NuuAdvhG7IojZSKnw Paper Abstract Text: We present new and improved fixed-parameter algo- rithms for computing maximum agreement forests (MAFs) of pairs of rooted binary phylogenetic trees. The size of such a forest for two trees corresponds to their subtree prune-and- regraft distance and, if the agreement forest is acyclic, to their hybridization number ... Attributes: – Authors: Chris Whidden, Robert G. Beiko, Norbert Zeh – Research Areas: q-bio.PE, cs.DS Figure 1: Examples of a Yelp review and an arXiv pa- per abstract and their corresponding attributes. Texts in typewriter font are attribute labels. (understanding language on its own, e.g., GLUE; Wang et al., 2018, inter alia), prior work has shown that incorporating these attributes into our model increases not just its …

Content truncated to 500 words