Pointing the Unknown Words

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 140–149, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Pointing the Unknown Words Caglar Gulcehre Universit´e de Montr´eal Sungjin Ahn Universit´e de Montr´eal Ramesh Nallapati IBM T.J. Watson Research Bowen Zhou IBM T.J. Watson Research Yoshua Bengio Universit´e de Montr´eal CIFAR Senior Fellow Abstract The problem of rare and unknown words is an important issue that can potentially effect the performance of many NLP sys- tems, including traditional count-based and deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models us- ing attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sen- tence, and the other predicts a word in the shortlist vocabulary. At each timestep, the decision of which softmax layer to use is adaptively made by an MLP which is con- ditioned on the context. We motivate this work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the envi- ronment when the name of an object is not known. Using our proposed model, we ob- serve improvements on two tasks, neural machine translation on the Europarl En- glish to French parallel corpora and text summarization on the Gigaword dataset. 1 Introduction Words are the basic input/output units in most of the NLP systems, and thus the ability to cover a large number of words is a key to building a ro- bust NLP system. However, considering that (i) the number of all words in a language including named entities is very large and that (ii) language itself is an evolving system (people create new words), this can be a challenging problem. A common approach followed by the recent neural network based NLP systems is to use a softmax output layer where each of the output di- mension corresponds to a word in a predefined word-shortlist. Because computing high dimen- sional softmax is computationally expensive, in practice the shortlist is limited to have only top- K most frequent words in the training corpus. All other words are then replaced by a special word, called the unknown word (UNK). The shortlist approach has two fundamental problems. The first problem, which is known as the rare word problem, is that some of the words in the shortlist occur less frequently in the train- ing set and thus are difficult to learn a good repre- sentation, resulting in poor performance. Second, it is obvious that we can lose some important in- formation by mapping different words to a single dummy token UNK. Even if we have a very large shortlist including all unique words in the training set, it does not necessarily improve the test perfor- mance, because there still exists a chance to see an unknown word at test time. This is known as the unknown …

Content truncated to 500 words