Probing via Prompting

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 107 - 115 July 10-15, 2022 ©2022 Association for Computational Linguistics Enhancing Self-Attention with Knowledge-Assisted Attention Maps Jiangang Bai1,∗, Yujing Wang2, Hong Sun2, Ruonan Wu2, Tianmeng Yang1, Pengfei Tang2, Defu Cao1,*, Mingliang Zhang1,*, Yunhai Tong1, Yaming Yang2, Jing Bai2, Ruofei Zhang2, Hao Sun2, Wei Shen2 1Peking University, Beijing, China 2Microsoft, Beijing, China {pku_bjg,youngtimmy,cdf,zml24,yhtong}@pku.edu.cn {yujwang,hosu,ruonan.wu,pengfeitang,yayaming,jbai,bzhang,hasun,sashen}@microsoft.com Abstract Large-scale pre-trained language models have attracted extensive attentions in the research community and shown promising results on var- ious tasks of natural language processing. How- ever, the attention maps, which record the at- tention scores between tokens in self-attention mechanism, are sometimes ineffective as they are learned implicitly without the guidance of explicit semantic knowledge. Thus, we aim to infuse explicit external knowledge into pre- trained language models to further boost their performance. Existing works of knowledge in- fusion largely depend on multi-task learning frameworks, which are inefficient and require large-scale re-training when new knowledge is considered. In this paper, we propose a novel and generic solution, KAM-BERT, which di- rectly incorporates knowledge-generated atten- tion maps into the self-attention mechanism. It requires only a few extra parameters and sup- ports efficient fine-tuning once new knowledge is added. KAM-BERT achieves consistent im- provements on various academic datasets for natural language understanding. It also out- performs other state-of-the-art methods which conduct knowledge infusion into transformer- based architectures. Moreover, we apply our model to an industry-scale ad relevance applica- tion and show its advantages in the real-world scenario. 1 Introduction Language models pre-trained by a large text cor- pus have shown superior performances on a wide range of natural language processing tasks. Many advanced models based on the transformer archi- tectures achieve state-of-the-art results on various NLP benchmarks. Existing literature (Jawahar et al., 2019; Hewitt and Manning, 2019) shows that pre-training enables a model to capture syntac- tic and semantic information in the self-attention mechanism. However, the attention maps, which ∗The work was done when the author visited Microsoft. record the attention scores between tokens in a self- attention mechanism, are sometimes ineffective as they are learned implicitly without the guidance of explicit semantics (Jain and Wallace, 2019). If the knowledge can be leveraged in a reasonable way to guide the self-attention mechanism, we have a good chance to improve the quality of attention scores as well as the performance of downstream applications. Recently, there have been multiple attempts for incorporating knowledge into transformer archi- tectures. ERNIE (Zhang et al., 2019) and KE- PLER (Wang et al., 2019) utilize both large-scale textual corpora and knowledge graphs to train a rep- resentation model in a multi-task learning frame- work. They need to be retrained from scratch when injecting new knowledge, which is inefficient and can not benefit from existing pre-trained check- points. K-Adapter (Wang et al., 2020) integrates additional neural models to capture different kinds of knowledge. It enables adaptation based on pre- trained language models. However, it does …

Content truncated to 500 words