A Mixture of h - 1 Heads is Better than h Heads

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 619–624 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 619 Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge Keqing He, Yuanmeng Yan, Weiran Xu ∗ Pattern Recognition & Intelligent System Laboratory School of Information and Communication Engineering Beijing University of Posts and Telecommunications, Beijing, China {kqin,yanyuanmeng,xuweiran}@bupt.edu.cn Abstract Neural-based context-aware models for slot tagging have achieved state-of-the-art perfor- mance. However, the presence of OOV(out- of-vocab) words significantly degrades the per- formance of neural-based models, especially in a few-shot scenario. In this paper, we pro- pose a novel knowledge-enhanced slot tagging model to integrate contextual representation of input text and the large-scale lexical back- ground knowledge. Besides, we use multi- level graph attention to explicitly model lexi- cal relations. The experiments show that our proposed knowledge integration mechanism achieves consistent improvements across set- tings with different sizes of training data on two public benchmark datasets. 1 Introduction Slot tagging is a critical component of spoken lan- guage understanding(SLU) in dialogue systems. It aims at parsing semantic concepts from user utter- ances. For instance, given the utterance ”I’d also like to have lunch during my flight” from the ATIS dataset, a slot tagging model might identify lunch as a meal description type. Given sufficient train- ing data, recent neural-based models (Mesnil et al., 2014; Liu and Lane, 2015, 2016; Goo et al., 2018; Haihong et al., 2019; He et al., 2020) have achieved remarkably good results. However, these works often suffer from poor slot tagging accuracy when rare words or OOV( out-of-vocab) words exist. (Ray et al., 2018) has verified the presence of OOV words further de- grades the performance of neural-based models, especially in a few-shot scenario where training data can not provide adequate contextual seman- tics. Previous context-aware models merely focus on how to capture deep contextual semantics to aid ∗ Weiran Xu is the corresponding author. playlist broadcast music genre classical music popular jazz scat singing hyponyms sister term train set can you append some classical music to my playlist O O O O B-music_type I-music_type O O O find and add some scat singing to my broadcast O O O O O O O test set semantic synset O O (context-aware model) B-music_type I-music_type (knowledge integration) Figure 1: An example of slot tagging in the few-shot scenario where scat singing is unseen in the training set. The prior context-aware model fails to recognize its correct type because of low-coverage contextual in- formation. After integrating background knowledge from WordNet, it succeeds to reason the correct type via lexical relations. in recognizing slot entities, while neglecting ontol- ogy behind the words or large-scale background knowledge. Explicit lexical relations are vital to recognizing unseen words when there is not ad- equate training data, that is, few-shot scenarios. Fig 1 gives a motivating example of slot tagging to explain the phenomenon. This example sug- gests slot tagging requires not only understanding the complex linguistic …

Content truncated to 500 words