Learning to Understand Phrases by Embedding the Dictionary

Learning to Understand Phrases by Embedding the Dictionary Felix Hill Computer Laboratory University of Cambridge felix.hill@cl.cam.ac.uk Kyunghyun Cho∗ Courant Institute of Mathematical Sciences and Centre for Data Science New York University kyunghyun.cho@nyu.edu Anna Korhonen Department of Theoretical and Applied Linguistics University of Cambridge alk23@cam.ac.uk Yoshua Bengio CIFAR Senior Fellow Universit´e de Montr´eal yoshua.bengio@umontreal.ca Abstract Distributional models that learn rich seman- tic word representations are a success story of recent NLP research. However, develop- ing models that learn useful representations of phrases and sentences has proved far harder. We propose using the definitions found in everyday dictionaries as a means of bridg- ing this gap between lexical and phrasal se- mantics. Neural language embedding mod- els can be effectively trained to map dictio- nary definitions (phrases) to (lexical) repre- sentations of the words defined by those defi- nitions. We present two applications of these architectures: reverse dictionaries that return the name of a concept given a definition or description and general-knowledge crossword question answerers. On both tasks, neural lan- guage embedding models trained on defini- tions from a handful of freely-available lex- ical resources perform as well or better than existing commercial systems that rely on sig- nificant task-specific engineering. The re- sults highlight the effectiveness of both neu- ral embedding architectures and definition- based training for developing models that un- derstand phrases and sentences. 1 Introduction Much recent research in computational seman- tics has focussed on learning representations of arbitrary-length phrases and sentences. This task is challenging partly because there is no obvious gold standard of phrasal representation that could be used ∗ Work mainly done at the University of Montreal. in training and evaluation. Consequently, it is diffi- cult to design approaches that could learn from such a gold standard, and also hard to evaluate or compare different models. In this work, we use dictionary definitions to ad- dress this issue. The composed meaning of the words in a dictionary definition (a tall, long-necked, spotted ruminant of Africa) should correspond to the meaning of the word they define (giraffe). This bridge between lexical and phrasal semantics is use- ful because high quality vector representations of single words can be used as a target when learning to combine the words into a coherent phrasal repre- sentation. This approach still requires a model capable of learning to map between arbitrary-length phrases and fixed-length continuous-valued word vectors. For this purpose we experiment with two broad classes of neural language models (NLMs): Recur- rent Neural Networks (RNNs), which naturally en- code the order of input words, and simpler (feed- forward) bag-of-words (BOW) embedding models. Prior to training these NLMs, we learn target lexi- cal representations by training the Word2Vec soft- ware (Mikolov et al., 2013) on billions of words of raw text. We demonstrate the usefulness of our approach by building and releasing two applications. The first is a reverse dictionary or concept finder: a system that returns words based on user descriptions or defini- tions (Zock and Bilac, 2004). Reverse dictionaries are …

Content truncated to 500 words