A Prism Module for Semantic Disentanglement in Name Entity Recognition

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5358–5362 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5358 A Prism Module for Semantic Disentanglement in Name Entity Recognition Kun Liu1,*, † Shen Li2,* Daqi Zheng2 Zhengdong Lu2 Sheng Gao1 Si Li1 {liukun, gaosheng, lisi}@bupt.edu.cn {shen, da, luz}@deeplycurious.ai 1 Beijing University of Posts and Telecommunications 2 Deeplycurious.ai Abstract Natural Language Processing has been per- plexed for many years by the problem that multiple semantics are mixed inside a word, even with the help of context. To solve this problem, we propose a prism module to dis- entangle the semantic aspects of words and re- duce noise at the input layer of a model. In the prism module, some words are selectively re- placed with task-related semantic aspects, then these denoised word representations can be fed into downstream tasks to make them eas- ier. Besides, we also introduce a structure to train this module jointly with the downstream model without additional data. This module can be easily integrated into the downstream model and significantly improve the perfor- mance of baselines on named entity recogni- tion (NER) task. The ablation analysis demon- strates the rationality of the method. As a side effect, the proposed method also provides a way to visualize the contribution of each word. 1 1 Introduction In Nature Language Processing (NLP), words con- tribute differently to different tasks. Therefore, attention-based models pay more attention on im- portant words than unimportant words. Since the information that is unrelated to the task can be re- garded as noise, unimportant words contain more noise than important words do. From this perspec- tive, attention is a noise reduction mechanism. Hard attention and soft attention are two main types of attention mechanisms which are proposed in (Xu et al., 2015). Hard attention mechanism se- lects some important tokens from input sequence ∗Kun Liu and Shen Li contributed equally to this work. †Work performed when Kun Liu worked as an intern in Deeplycurious.ai. 1Our code is available at https://github.com/ liukun95/Prism-Module and ignore others. This will lead to the loss of necessary information which exists in the ignored tokens. By contrast, in soft attention mechanism, a probability distribution which reflects the impor- tance of tokens is calculated over each token of the input sequence. However, since there is more use- less information than useful information in unim- portant words, it should be noted that noise could be kept more, when those words are assigned with non-zero probabilities. Overall, both two attention mechanisms have drawbacks in noise reduction. Attention mechanism is firstly applied in Com- puter Vision (CV) (Mnih et al., 2014) where pixels are the basic units. However, in NLP, the mini- mum unit is not word but sense. Therefore, NLP tasks need a noise reduction method at a finer granularity than attention mechanism. Normally, various aspects of semantics are en- tangled in word embeddings (Bengio et al., 2003; Mikolov et al., 2013). However, only some of …

Content truncated to 500 words