VCDM: Leveraging Variational Bi-encoding and Deep Contextualized Word Representations for Improved Definition Modeling

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6331–6344, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6331 VCDM: Leveraging Variational Bi-encoding and Deep Contextualized Word Representations for Improved Definition Modeling Machel Reid Edison Marrese-Taylor Graduate School of EngineeringYutaka Matsuo The University of Tokyo machelreid2004@gmail.com {emarrese, matsuo}@weblab.t.u-tokyo.ac.jp Abstract In this paper we tackle the task of definition modeling, where the goal is to learn to gen- erate definitions of words and phrases. Exist- ing approaches for this task are discriminative, combining distributional and lexical semantics in an implicit rather than direct way. To tackle this issue we propose a generative model for the task, introducing a continuous latent vari- able to explicitly model the underlying rela- tionship between a phrase used within a con- text and its definition. We rely on variational inference for estimation and leverage contex- tualized word embeddings for improved per- formance. Our approach is evaluated on four existing challenging benchmarks with the ad- dition of two new datasets, CAMBRIDGE and the first non-English corpus ROBERT, which we release to complement our empirical study. Our Variational Contextual Definition Mod- eler (VCDM) achieves state-of-the-art perfor- mance in terms of automatic and human evalu- ation metrics, demonstrating the effectiveness of our approach.1 1 Introduction In most current NLP tasks, fixed-length vector rep- resentations of words, word embeddings, are used to represent some form of the meaning of the word. In the case of humans, however, oftentimes we will use a sequence of words known as a definition —a statement of the meaning for a term— to express meanings of terms (words, phrases, or symbols). It is with this in mind that the question of “Can ma- chines define?” is aimed to be answered with the task of definition modeling (Noraset et al., 2017). Definition modeling can be framed as a task of conditional generation, in which the definition d of the word or phrase is generated given a con- ditioning variable w such as a word’s associated 1We release the code at: https://github.com/ machelreid/vcdm word embedding or other representations of con- text. Current approaches for this task (Noraset et al., 2017; Gadetsky et al., 2018; Ni and Wang, 2017; Ishiwatari et al., 2019) are mainly encoder- decoder based, in which one encodes a contextual representation for a word/phrase w using a variety of features such as context or character composi- tion, and uses the contextual representation(s) to generate the definition d. Despite the relative success of existing approaches for definition modelling, their discriminative na- ture —where distributional-derived information is at one end of the model and lexical information is at the other— limits their power as the underly- ing semantic representations of the distributional and lexical information are learned in an implicit rather than direct way. For example, although Ishi- watari et al. (2019) successfully showed that both local and global contexts are useful to disambiguate meanings of phrases in certain cases, their approach heavily relies on an attention mechanism to identify semantic alignments …

Content truncated to 500 words