Adaptive Threshold Selective Self-Attention for Chinese NER

Proceedings of the 29th International Conference on Computational Linguistics, pages 1823–1833 October 12–17, 2022. 1823 Adaptive Threshold Selective Self-Attention for Chinese NER Biao Hu1, Zhen Huang1∗, Minghao Hu2∗, Ziwen Zhang1, Yong Dou1 1College of Computer, National University of Defense Technology, Changsha, China 2Information Research Center of Military Science, PLA Academy of Military Science, Beijing, China {hubiao, huangzhen, ziwen, yongdou}@nudt.edu.cn huminghao16@gmail.com Abstract Recently, Transformer has achieved great suc- cess in Chinese named entity recognition (NER) owing to its good parallelism and abil- ity to model long-range dependencies, which utilizes self-attention to encode context. How- ever, the fully connected way of self-attention may scatter the attention distribution and al- low some irrelevant character information to be integrated, leading to entity boundaries be- ing misidentified. In this paper, we propose a data-driven Adaptive Threshold Selective Self- Attention (ATSSA) mechanism that aims to dy- namically select the most relevant characters to enhance the Transformer architecture for Chinese NER. In ATSSA, the attention score threshold of each query is automatically gener- ated, and characters with attention score higher than the threshold are selected by the query while others are discarded, so as to address ir- relevant attention integration. Experiments on four benchmark Chinese NER datasets show that the proposed ATSSA brings 1.68 average F1 score improvements to the baseline model and achieves state-of-the-art performance. 1 Introduction Named Entity Recognition (NER) aims to identify named entities in the given text, including persons, locations, organizations, etc. It plays an impor- tant role in downstream natural language process- ing (NLP) tasks such as information retrieval (Chen et al., 2015), relation extraction (Miwa and Bansal, 2016) and question answering (Diefenbach et al., 2018). Compared with English NER, Chinese NER is more difficult since there is no natural delimiter between words in Chinese sentences. To integrate word information related to each character and avoid error propagation of word seg- mentation, word-character lattice is first applied to Chinese NER in Lattice-LSTM (Zhang and Yang, 2018). However, the RNN-based model is ∗Corresponding author 海 大 众 文 化 艺 术 节 上 Selective Self-Attention 海 大 众 文 化 艺 术 节 Fully Connected 上 Self-Attention Shanghai popular culture and art festival Figure 1: A comparison of selective self-attention and fully connected self-attention, where the attention weights are indicated by blue color. We only show the attention distribution of the character "上 (Shang)" in the head that focuses on entities. hard to model long-range dependencies, and has poor computational efficiency. Recently, Trans- former (Vaswani et al., 2017) is widely used in Chinese NER, which utilizes self-attention to en- code context. Since vanilla Transformer lacks di- rectionality, TENER (Yan et al., 2019) combines Transformer encoder with direction-aware position encoding to catch directions between characters. FLAT (Li et al., 2020) further exploits span relative position encoding to convert the lattice structure into a flat structure. Although Transformer-based models perform well on Chinese NER, they have a limitation that the fully connected self-attention drives queries to attend to all characters of the inputs, making irrel- …

Content truncated to 500 words