DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2679–2693 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2679 DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling Baojun Wang1∗, Zhao Zhang2∗, Kun Xu2∗, Guang-Yuan Hao3, Yuyang Zhang1 Lifeng Shang1, Linlin Li2, Xiao Chen1, Xin Jiang1, Qun Liu1 1Huawei Noah’s Ark Lab 2Huawei Technologies Co., Ltd. 3The Hong Kong University of Science and Technology {puking.w,zhangzhao54,xukun24,zhangyuyang4}@huawei.com {Shang.Lifeng,lynn.lilinlin,chen.xiao2,Jiang.Xin,qun.liu}@huawei.com guangyuanhao@outlook.com Abstract Incorporating lexical knowledge into deep learning models has been proved to be very ef- fective for sequence labeling tasks. However, previous works commonly have difficulty deal- ing with large-scale dynamic lexicons which often cause excessive matching noise and prob- lems of frequent updates. In this paper, we propose DyLex, a plug-in lexicon incorpora- tion approach for BERT based sequence la- beling tasks. Instead of leveraging embed- dings of words in the lexicon as in conven- tional methods, we adopt word-agnostic tag embeddings to avoid re-training the representa- tion while updating the lexicon. Moreover, we employ an effective supervised lexical knowl- edge denoising method to smooth out match- ing noise. Finally, we introduce a col-wise attention based knowledge fusion mechanism to guarantee the pluggability of the proposed framework. Experiments on ten datasets of three tasks show that the proposed framework achieves new SOTA, even with very large scale lexicons1. 1 Introduction Sequence labeling is the task of assigning cate- gorical labels to a text sequence. Many conven- tional NLP tasks, such as named entity recogni- tion (NER), Chinese word segmentation (CWS), and slot-filling based natural language understand- ing (NLU), can be formalized as the sequence la- beling problem. The deep learning methods, espe- cially the recently proposed BERT and its variants, have achieved great success in such sequence la- beling tasks. However, the BERT-based methods are generally built based on word-piece or charac- ter embeddings. The word information (e.g., word boundary or type) is not fully exploited, which makes it difficult to accurately determine the entity boundary or correctly predict entity type. *Equal contribution. 1https://github.com/huawei-noah/ noah-research/Dylex please open the IronMan play Just a Little While Longer now on IronMan what is IronMan music?movie? ambiguous and long name Figure 1: Iron Man can be a name of a smart device or a movie and the system would be unable to react properly upon “Please play Iron Man” from a user. An- other case as “Play just a little while longer now on Iron Man” requires the system to classify “Play” be- tween music and movie domains, and whether “now” should be combined with “just a little while longer” as a whole. As shown in Figure 1, it is infeasible to under- stand user’s utterance correctly without using de- terministic domain knowledge that “Iron Man” is the alias of a Smart Speaker or “just a little while longer” is a famous song. In commercial systems, the lexicon is widely used as an effective way to store various domain knowledge. In practice, the size of a lexicon can range from ten to …

Content truncated to 500 words