Exploring Kernel Functions in the Softmax Layer for Contextual Word Classification

Exploring Kernel Functions in the Softmax Layer for Contextual Word Classification Yingbo Gao, Christian Herold, Weiyue Wang, Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany <surname>@i6.informatik.rwth-aachen.de Abstract Prominently used in support vector machines and logistic re- gressions, kernel functions (kernels) can implicitly map data points into high dimensional spaces and make it easier to learn complex decision boundaries. In this work, by replac- ing the inner product function in the softmax layer, we ex- plore the use of kernels for contextual word classification. In order to compare the individual kernels, experiments are con- ducted on standard language modeling and machine transla- tion tasks. We observe a wide range of performances across different kernel settings. Extending the results, we look at the gradient properties, investigate various mixture strategies and examine the disambiguation abilities. 1. Introduction With neural networks, tasks such as language modeling (LM) and machine translation (MT) are generally approached by factorizing the target sentence probability into products of target word posterior probabilities [1, 2, 3]. In order to clas- sify over the target vocabulary, it is necessary to compute a context vector, learn a projection matrix and normalize the similarity scores between the two probabilities. While vari- ous model architectures are proposed to calculate the context vectors [4, 5, 6], most of them use a softmax layer with the in- ner product function to compute the word posterior probabil- ities. [7] identify a shortcoming with the formulation above which they call the “Softmax Bottleneck”. The problem lies in the exponential-and-logarithm calculation when using the cross-entropy criterion, which results in a low-rank log word posterior probability matrix. Hypothesizing that natural lan- guage is high-rank, the authors argue that the “Softmax Bot- tleneck” is a limiting factor of the expressiveness of the mod- els. One natural thought on this problem is its similarity to the lack of expressiveness of a logistic regression model or a support vector machine (SVM) with a simple linear kernel. By implicitly transforming data points into high dimen- sional feature spaces, kernels can increase the expressiveness of the classifier and allow for more complex decision bound- aries [8]. Note that a kernel is deemed valid when it corre- sponds to a scalar product in some feature space [8], or its corresponding Gram matrix is positive semidefinite [9]. Yet empirical results [10, 11] also show that conditionally posi- tive semidefinite kernels can perform well in some applica- tions. In this work, we do not enforce the positive semidefi- niteness of kernels. Motivated to examine the performances of various ker- nels in LM and MT, we structure this work as follows: 1. We implement individual kernels in replacement of the inner product function in the softmax layer and test them on LM and MT tasks. 2. We look at the gradient properties of several kernels and analyze the observed performance differences. 3. We investigate various mixtures of kernels. 4. We further examine and compare the disambiguation abilities of the …

Content truncated to 500 words