Deep Generative Model for Joint Alignment and Word Representation

Proceedings of NAACL-HLT 2018, pages 1011–1023 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Deep Generative Model for Joint Alignment and Word Representation Miguel Rios Wilker Aziz Khalil Sima’an Institute for Logic, Language, and Computation University of Amsterdam {m.riosgaona, w.aziz, k.simaan}@uva.nl Abstract This work exploits translation data as a source of semantically relevant learning signal for models of word representation. In particular, we exploit equivalence through translation as a form of distributional context and jointly learn how to embed and align with a deep gener- ative model. Our EMBEDALIGN model em- beds words in their complete observed context and learns by marginalisation of latent lexical alignments. Besides, it embeds words as pos- terior probability densities, rather than point estimates, which allows us to compare words in context using a measure of overlap between distributions (e.g. KL divergence). We inves- tigate our model’s performance on a range of lexical semantics tasks achieving competitive results on several standard benchmarks includ- ing natural language inference, paraphrasing, and text similarity. 1 Introduction Natural language processing applications often count on the availability of word representations trained on large textual data as a means to alleviate problems such as data sparsity and lack of linguis- tic resources (Collobert et al., 2011; Socher et al., 2011; Tu et al., 2017; Bowman et al., 2015). Traditional approaches to inducing word repre- sentations circumvent the need for explicit seman- tic annotation by capitalising on some form of in- direct semantic supervision. A typical example is to fit a binary classifier to detect whether or not a target word is likely to co-occur with neighbour- ing words (Mikolov et al., 2013). If the binary classifier represents a word as a continuous vector, that vector will be trained to be discriminative of the contexts it co-occurs with, and thus words in similar contexts will have similar representations. Code available from https://github.com/ uva-slpl/embedalign MR and WA contributed equally. The underlying assumption is that context (e.g. neighbouring words) stands for the meaning of the target word (Harris, 1954; Firth, 1957). The suc- cess of this distributional hypothesis hinges on the definition of context and different models are based on different definitions. Importantly, the nature of the context determines the range of linguistic properties the representations may capture (Levy and Goldberg, 2014b). For example, Levy and Goldberg (2014a) propose to use syntactic context derived from dependency parses. They show that their representations are much more discriminative of syntactic function than models based on imme- diate neighbourhood (Mikolov et al., 2013). In this work, we take lexical translation as indi- rect semantic supervision (Diab and Resnik, 2002). Effectively we make two assumptions. First, that every word has a foreign equivalent that stands for its meaning. Second, that we can find this equiva- lent in translation data through lexical alignments.1 For that we induce both a latent mapping between words in a bilingual sentence pair and distributions over latent word representations. To summarise our contributions: • we model a joint distribution …

Content truncated to 500 words