Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 337–348 November 10–11, 2021. ©2021 Association for Computational Linguistics 337 Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning Christos Theodoropoulos KU Leuven christos.theodoropoulos@kuleuven.be James Henderson Idiap Research Institute james.henderson@idiap.ch Andrei Catalin Coman EPFL, Idiap Research Institute andrei.coman@idiap.ch Marie-Francine Moens KU Leuven sien.moens@kuleuven.be Abstract Though language model text embeddings have revolutionized NLP research, their ability to capture high-level semantic information, such as relations between entities in text, is limited. In this paper, we propose a novel contrastive learning framework that trains sentence em- beddings to encode the relations in a graph structure. Given a sentence (unstructured text) and its graph, we use contrastive learning to impose relation-related structure on the token- level representations of the sentence obtained with a CharacterBERT (El Boukkouri et al., 2020) model. The resulting relation-aware sen- tence embeddings achieve state-of-the-art re- sults on the relation extraction task using only a simple KNN classifier, thereby demonstrat- ing the success of the proposed method. Addi- tional visualization by a tSNE analysis shows the effectiveness of the learned representation space compared to baselines. Furthermore, we show that we can learn a different space for named entity recognition, again using a con- trastive learning objective, and demonstrate how to successfully combine both representa- tion spaces in an entity-relation task. 1 Introduction Pretrained language models (LMs), such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and GPT-3 (Brown et al., 2020), capture contex- tualized information effectively and are used in a wide variety of natural language processing (NLP) tasks. They have revolutionized NLP research. The main mechanism of these models is multi-head self-attention (Vaswani et al., 2017), which enables capturing patterns of semantic and syntactic inter- est in text. However, their ability to encapsulate high level semantic information, such as relations in the text, and domain-specific knowledge, is lim- ited because they are trained on very large corpora using the main objectives of language modeling. In many NLP tasks, pretrained LM embeddings are used as model input. A common strategy is to concatenate the embeddings that are extracted from different LMs and let the model decide which part of the information is useful for the task. This empirical approach does not provide strong intu- ition and results in poor explainability capabilities because most of the task-specific models are black boxes. In this study, we present a novel contrastive learn- ing (CL) framework to leverage the embedding space of CharacterBERT and impose a relation structure on the embeddings. The proposed frame- work receives a sentence and a graph that repre- sents the text relations in a structured way, and the CL paradigm is applied to impose this struc- ture on the token embeddings of the Character- BERT text encoder. Different graph formulations that represent the text relations are explored. The main goal is to create a common embedding space where relations can be easily detected. To evaluate progress towards this goal, we use the ADE dataset (Gurulingappa …

Content truncated to 500 words