Improving Natural Language Understanding by Reverse Mapping Bytepair Encoding

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 163–173 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 163 Improving Natural Language Understanding by Reverse Mapping Bytepair Encoding Chaodong Tong1,2 Huailiang Peng1,2(�) Qiong Dai1,2 Lei Jiang1,2 Jianghua Huang3 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3Meituan-Dianping Group, Beijing, China {tongchaodong, penghuailiang, daiqiong, jianglei}@iie.ac.cn {huangjianghua}@meituan.com Abstract Recently, language models (LMs) or language representation models are widely used in nat- ural language understanding (NLU) tasks. However, these LMs are usually trained on large unlabeled text corpora, while the fine- tuning process simply takes words or word- pieces as model input. Because of the differ- ences between language model and NLU task objectives, the problem of lack of concern on some key words exists. Thus in this paper, we propose a method called reverse mapping bytepair encoding, which maps named-entity information and other word-level linguistic features back to subwords during the encod- ing procedure of bytepair encoding (BPE). We employ this method to the Generative Pre-trained Transformer (OpenAI GPT) (Rad- ford et al., 2018) by adding a weighted linear layer after the embedding layer. We also pro- pose a new model architecture named as the multi-channel separate transformer to evalu- ate the effectiveness of the newly introduced information by employing a training process without parameter-sharing. Experiments on Story Cloze, RTE, SciTail and SST-2 datasets demonstrate the effectiveness of our approach. Compared with the original results in GPT, our approach gains 1.58% absolute increase on Stories Cloze, 6.4% on RTE, 0.69% on SciTail and 0.8% on SST-2. 1 Introduction Recently, language models are widely used as the feature extractor for many NLU tasks. Statistical language models learn the joint probability func- tion of sequences of words in a language (Bengio et al., 2003). Trained on large corpora and differ- ent domains give LMs generalization abilities, and enable them to capture latent or deep senmantics. Normally, statistical language models take the fol- lowing objective to maximize: L1(u) = � i logP(ui|ui−k, ..., ui−1; Θ) (1) where k is the size of the context window, and the conditional probability P is modeled using a neu- ral network with parameters Θ. However, this formal simplicity determines that they can not deal well with the ambiguity of words nor low-frequency words. In fact, low-frequency words may appear once or little times in the whole corpus, thus the embeddings of them may over- fit or underfit the corpus. In practice, we usu- ally truncate them from the vocabulary and re- place them with an “UNK” label. In this situ- ation, we partially lose their meanings. For ex- ample, in “15 million tonnes of rubbish are pro- duced daily in Cairo.”, the number “15” can hardly be trained to gain its proper representation even if it is replaced by a specific label representing numbers. Besides, out of vocabulary (OOV) is a big problem while predicting. Some kinds of word segmentation …

Content truncated to 500 words