Customized Attention Mechanism for Relation Classification

Customized Attention Mechanism for Relation Classification Shirong Shen † Yang Wen † School of Computer Science and Engineer- ing, University of Electronic Science and Technology of China School of Computer Science and Engineer- ing, University of Electronic Science and Technology of China Chengdu, China Chengdu, China 1170743490@qq.com 201621060713@std.uestc.edu.cn Lijuan Zhou Hongyin Zan Natural Language Processing Laboratory, College of Information and Engineering, Zhengzhou University Natural Language Processing Laboratory, College of Information and Engineering, Zhengzhou University Zhengzhou, China Zhengzhou, China 1967331775@qq.com iehyzan@zzu.edu.cn Abstract This paper proposes a novel attention mecha- nism, customized attention mechanism, for rela- tion classification. Based on the task require- ments, logical constraints of the attention mech- anism have been designed to control the learning process of network, which enhances the control- lability in learning. Specifically, a logical con- straint is an operation that affects the attention weight. This operation controls the weights of the attention mechanism through a generating function and a limiting function. Appropriate logical constraints improve the feature extrac- tion ability and the robustness of the network, which are important supplements to the tradi- tional neural network. Experiments show that our model and logical constraints have achieved a considerable improvement on the SemEval- 2010 relation classification task, which outper- forms the most advanced methods. 1 Introduction Relational classification is an important topic of nat- ural language processing. It aims to extract the se- mantic relationship between two entities in a sen- tence. For instance, in the sentence “[brandy] is dis- tilled from fermented fruit [juices]”, the entities ‘brandy’ and ‘juices’ are of relation Entity-Origin. Relational classification can be applied to semantic information extraction, questions answering and knowledge-based completion. Traditional relation classification methods empha- size well-designed features (Rink and Harabagiu, 2010), or extend features using the kernel (Zelenko et al., 2003; Bunescu and Mooney, 2005), Rink and Harabagiu (2010) have achieved the best results by using support vector machines at the early stages. However, such methods are highly dependent on hand-craft linguistic features, which result in signif- icant limitations and numerous preprocessing effort. Nowadays, deep neural networks have achieved great success in relational classification task. A number of methods have been proposed for relation classification which use two architectures including convolutional neural networks (CNN) and recurrent neural networks (RNN). Socher et al. (2012) used an RNN-based architecture to generate composi- tional vector representations of sentences. Zeng et al. (2014) proposed a CNN network integrating with position embeddings to make up for the shortcom- ings of CNN missing contextual information. How- ever, the features extracted by simple RNN and CNN are still not sensitive to location information which may miss core semantics. Thus, the attention mechanism has been designed to solve this problem. PACLIC 32 591 32nd Pacific Asia Conference on Language, Information and Computation Hong Kong, 1-3 December 2018 Copyright 2018 by the authors Zhou et al. (2016) applied attention after bi-direc- tional recurrent neural networks to extract relations between two entities. A multi-level attention CNN was proposed by Wang et al. (2016). Although such methods have …

Content truncated to 500 words