Attention-based Conditioning Methods for External Knowledge Integration

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3944–3951 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3944 Attention-based Conditioning Methods for External Knowledge Integration Katerina Margatina1, Christos Baziotis2 ∗, Alexandros Potamianos1,3,4 1School of ECE, National Technical University of Athens, Athens, Greece 2 School of Informatics, University of Edinburgh, UK 3 Signal Analysis and Interpretation Laboratory (SAIL), USC, Los Angeles, USA 4 Behavioral Signal Technologies, Los Angeles, USA el12108@central.ntua.gr, c.baziotis@sms.ed.ac.uk, potam@central.ntua.gr Abstract In this paper, we present a novel approach for incorporating external knowledge in Re- current Neural Networks (RNNs). We pro- pose the integration of lexicon features into the self-attention mechanism of RNN-based architectures. This form of conditioning on the attention distribution, enforces the contri- bution of the most salient words for the task at hand. We introduce three methods, namely attentional concatenation, feature-based gating and affine transformation. Experiments on six benchmark datasets show the effectiveness of our methods. Attentional feature-based gating yields consistent performance improvement across tasks. Our approach is implemented as a simple add-on module for RNN-based mod- els with minimal computational overhead and can be adapted to any deep neural architecture. 1 Introduction Modern deep learning algorithms often do away with feature engineering and learn latent represen- tations directly from raw data that are given as in- put to Deep Neural Networks (DNNs) (Mikolov et al., 2013; McCann et al., 2017; Peters et al., 2018). However, it has been shown that linguistic knowledge (manually or semi-automatically en- coded into lexicons and knowledge bases) can sig- nificantly improve DNN performance for Natural Language Processing (NLP) tasks, such as natural language inference (Mrkši´c et al., 2017), language modelling (Ahn et al., 2016), named entity recog- nition (Ghaddar and Langlais, 2018) and relation extraction (Vashishth et al., 2018). For NLP tasks, external sources of information are typically incorporated into deep neural archi- tectures by processing the raw input in the context of such external linguistic knowledge. In machine ∗ The research was conducted when the author was a re- searcher at School of ECE, NTUA in Athens, Greece. learning, this contextual processing is known as conditioning; the computation carried out by a model is conditioned or modulated by informa- tion extracted from an auxiliary input. The most commonly-used method of conditioning is con- catenating a representation of the external infor- mation to the input or hidden network layers. Attention mechanisms (Bahdanau et al., 2015; Vaswani et al., 2017; Lin et al., 2017) are a key ingredient for achieving state-of-the-art perfor- mance in tasks such as textual entailment (Rock- täschel et al., 2016), question answering (Xiong et al., 2017), and neural machine translation (Wu et al., 2016). Often task-specific attentional archi- tectures are proposed in the literature to further improve DNN performance (Dhingra et al., 2017; Xu et al., 2015; Barrett et al., 2018). In this work, we propose a novel way of utiliz- ing word-level prior information encoded in lin- guistic, sentiment, and emotion lexicons, to im- prove classification …

Content truncated to 500 words