MA-BERT: Learning Representation by Incorporating Multi-Attribute Knowledge in Transformers

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2338–2343 August 1–6, 2021. ©2021 Association for Computational Linguistics 2338 MA-BERT: Learning Representation by Incorporating Multi-Attribute Knowledge in Transformers You Zhang†, Jin Wang†, 1, Liang-Chih Yu‡, 2 and Xuejie Zhang†, 3 †School of Information Science and Engineering, Yunnan University, Yunnan, P.R. China ‡Department of Information Management, Yuan Ze University, Taiwan Contact:{wangjin1, xjzhang3}@ynu.edu.cn, lcyu@saturn.yzu.edu.tw2 Abstract Incorporating attribute information such as user and product features into deep neural net- works has been shown to be useful in senti- ment analysis. Previous works typically ac- complished this in two ways: concatenating multiple attributes to word/text representation or treating them as a bias to adjust attention dis- tribution. To leverage the advantages of both methods, this paper proposes a multi-attribute BERT (MA-BERT) to incorporate external at- tribute knowledge. The proposed method has two advantages. First, it applies multi-attribute transformer (MA-Transformer) encoders to in- corporate multiple attributes into both input representation and attention distribution. Sec- ond, the MA-Transformer is implemented as a universal layer and stacked on a BERT-based model such that it can be initialized from a pre-trained checkpoint and fine-tuned for the downstream applications without extra pre- training costs. Experiments on three bench- mark datasets show that the proposed method outperformed pre-trained BERT models and other methods incorporating external attribute knowledge. 1 Introduction To learn a distributed text representation for sen- timent classification (Pang and Lee, 2008; Liu, 2012), conventional deep neural networks, such as convolutional neural networks (CNN) (Kim, 2014) and long short-term memory (LSTM) (Hochre- iter and Schmidhuber, 1997), and common in- tegration technics, such as self-attention mecha- nisms (Vaswani et al., 2017; Chaudhari et al., 2019) and dynamic routing algorithms (Gong et al., 2018; Sabour et al., 2017), are usually applied to com- pose the vectors of constituent words. To further en- hance the performance, pre-trained models (PTMs), such as BERT (Devlin et al., 2019), ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), and XLM- Text Attr Word Embeddings Feature Extractor (a) Word representation Text Attr Word Embeddings Feature Extractor (b) Text representation Attribute Embeddings Attribute Embeddings Text Attr Word Embeddings Feature Extractor Attribute Embeddings Attention map (c) Bias terms in self-attention (d) Bilinear interaction in multi-attribute transformer Text Attr Word Embeddings Attribute Embeddings Q K V Scaled Dot-Product Attention Figure 1: Different strategies to incorporate external attribute knowledge into deep neural networks. RoBERTa (Conneau et al., 2019) can be fine-tuned and transferred for sentiment analysis tasks. Practi- cally, PTMs were first fed a large amount of unan- notated data, and trained using a masked language model or next sentence prediction to learn the usage of various words and how the language is written in general. Then, the models are transferred to an- other task to be fed another smaller task-specific dataset. The abovementioned methods only use features from plain texts. Incorporating attribute informa- tion such as users and products can improve senti- ment analysis task performance. Previous works typically incorporated such external knowledge by concatenating these attributes …

Content truncated to 500 words