Reliability-aware Dynamic Feature Composition for Name Tagging

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 165–174 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 165 Reliability-aware Dynamic Feature Composition for Name Tagging Ying Lin1, Liyuan Liu2, Heng Ji1,2, Dong Yu3 and Jiawei Han2 1 Dept. of Computer Science, Rensselaer Polytechnic Institute, Troy, NY, USA 2 Dept. of Computer Science, University of Illinois at Urbana-Champaign Urbana, IL, USA {yinglin8,ll2,hengji,hanj}@illinois.edu 3 Tencent AI Lab, Bellevue, WA, USA dyu@tencent.com Abstract While word embeddings are widely used for a variety of tasks and substantially improve the performance, their quality is not consis- tent throughout the vocabulary due to the long- tail distribution of word frequency. Without sufficient contexts, embeddings of rare words are usually less reliable than those of com- mon words. However, current models typ- ically trust all word embeddings equally re- gardless of their reliability and thus may in- troduce noise and hurt the performance. Since names often contain rare and unknown words, this problem is particularly critical for name tagging. In this paper, we propose a novel reliability-aware name tagging model to tackle this issue. We design a set of word frequency- based reliability signals to indicate the quality of each word embedding. Guided by the re- liability signals, the model is able to dynami- cally select and compose features such as word embedding and character-level representation using gating mechanisms. For example, if an input word is rare, the model relies less on its word embedding and assigns higher weights to its character and contextual features. Experi- ments on OntoNotes 5.0 show that our model outperforms the baseline model, obtaining up to 6.2% absolute gain in F-score. In cross- genre experiments on six genres in OntoNotes, our model improves the performance for most genre pairs and achieves 2.3% absolute F- score gain on average. 1 1 Introduction Serving as the basic unit of the model input, word embeddings form the foundation of various nat- ural language processing techniques using deep neural networks. Embeddings can effectively en- code semantic information and have proven suc- cessful in a wide range of tasks, such as sequence 1Code and resources for this paper: https://github. com/limteng-rpi/neural_name_tagging A MedChem spokesman said the products contribute about a third of MedChem's sales and 10% to 20% of its earnings MedChem spokesman said ... Word Embedding LSTM Encoder ORG O O CRF Context-only Features Rare word. Its embedding is unreliable. Rely more on surface and context clues. Common word. Its word embedding should be reliable. Reliability Signals Character-level Representation Gate Figure 1: A simplified illustration of the proposed model. We only show the backward part in the figure. labeling (Collobert et al., 2011; Chiu and Nichols, 2016; Ma and Hovy, 2016; Lample et al., 2016), text classification (Tang et al., 2014; Lai et al., 2015; Yang et al., 2016), and parsing (Chen and Manning, 2014; Dyer et al., 2015). Still, due to the long tail distribution, the quality of pre-trained word embeddings is usually inconsistent. Without sufficient …

Content truncated to 500 words