Learning Task-specific Bilexical Embeddings

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 161–171, Dublin, Ireland, August 23-29 2014. Learning Task-specific Bilexical Embeddings Pranava Swaroop Madhyastha Xavier Carreras Ariadna Quattoni TALP Research Center Universitat Polit`ecnica de Catalunya Campus Nord UPC, Barcelona pranava,carreras,aquattoni@lsi.upc.edu Abstract We present a method that learns bilexical operators over distributional representations of words and leverages supervised data for a linguistic relation. The learning algorithm exploits low- rank bilinear forms and induces low-dimensional embeddings of the lexical space tailored for the target linguistic relation. An advantage of imposing low-rank constraints is that prediction is expressed as the inner-product between low-dimensional embeddings, which can have great computational benefits. In experiments with multiple linguistic bilexical relations we show that our method effectively learns using embeddings of a few dimensions. 1 Introduction We address the task of learning functions that compute compatibility scores between pairs of lexical items under some linguistic relation. We refer to these functions as bilexical operators. As an instance of this problem, consider learning a model that predicts the probability that an adjective modifies a noun in a sentence. In this case, we would like the bilexical operator to capture the fact that some adjectives are more compatible with some nouns than others. For example, a bilexical operator should predict that the adjective electronic has high probability of modifying the noun device but little probability of modifying the noun case. Bilexical operators can be useful for multiple NLP applications. For example, they can be used to reduce ambiguity in a parsing task. Consider the following sentence extracted from a weblog: Vynil can be applied to electronic devices and cases, wooden doors and furniture and walls. If we want to predict the dependency structure of this sentence we need to make several decisions. In particular, the parser would need to decide (1) Does electronic modify devices? (2) Does electronic modify cases? (3) Does wooden modify doors? (4) Does wooden modify furniture? Now imagine that in the corpus used to train the parser none of these nouns have been observed, then it is unlikely that these attachments can be resolved correctly. However, if an accurate noun-adjective bilexical operator were available most of the uncertainty could be resolved. This is because a good bilinear operator would give high probability to the pairs electronic-device, wooden-door, wooden-furniture and low probability to the pair electronic-case. The simplest way of inducing a bilexical operator is to learn it from a training corpus. That is, assuming that we are given some data annotated with a linguistic relation between a modifier and a head (e.g. adjective and noun) we can simply build a maximum likelihood estimator for Pr(m | h) by counting the occurrences of modifiers and heads under the target relation. For example, we could consider learning bilexical operators from sentences annotated with dependency structures. Clearly, this model can not generalize to head words not present in the training data. To mitigate this we could consider bilexical operators that can exploit lexical embeddings, such …

Content truncated to 500 words