CoLAKE: Contextualized Language and Knowledge Embedding

Proceedings of the 28th International Conference on Computational Linguistics, pages 3660–3670 Barcelona, Spain (Online), December 8-13, 2020 3660 CoLAKE: Contextualized Language and Knowledge Embedding Tianxiang Sun1,∗, Yunfan Shao1, Xipeng Qiu1,† Qipeng Guo1, Yaru Hu1, Xuanjing Huang1, Zheng Zhang2 1Shanghai Key Laboratory of Intelligent Information Processing, Fudan University 1School of Computer Science, Fudan University 2Amazon Shanghai AI Lab {txsun19,yfshao19,xpqiu,qpguo16,xjhuang}@fudan.edu.cn yrhu112358@outlook.com zhaz@amazon.com Abstract With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the performance gains of these models. Few works explore the poten- tial of deep contextualized knowledge representation when injecting knowledge. In this paper, we propose the Contextualized Language and Knowledge Embedding (CoLAKE), which jointly learns contextualized representation for both language and knowledge with the extended MLM objective. Instead of injecting only entity embeddings, CoLAKE extracts the knowledge context of an entity from large-scale knowledge bases. To handle the heterogeneity of knowledge con- text and language context, we integrate them in a unified data structure, word-knowledge graph (WK graph). CoLAKE is pre-trained on large-scale WK graphs with the modified Transformer encoder. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks. Experimental results show that CoLAKE outperforms previous counterparts on most of the tasks. Besides, CoLAKE achieves surprisingly high performance on our synthetic task called word-knowledge graph completion, which shows the superiority of simultaneously contextualizing language and knowledge representation.1 1 Introduction Deep contextualized language models pre-trained on large-scale unlabeled corpora have achieved signif- icant improvement on a wide range of NLP tasks (Peters et al., 2018; Devlin et al., 2019; Yang et al., 2019). However, they are shown to have difficulty capturing factual knowledge (Logan et al., 2019). Recently, there is a growing interest in combining pre-trained language models (PLMs) with structured knowledge. A popular approach is to inject pre-trained entity embeddings into PLMs to better capture factual knowledge, such as ERNIE (Zhang et al., 2019) and KnowBERT (Peters et al., 2019). The shortcomings of these models can be summarized as follows: (1) The entity embeddings are separately pre-trained with some knowledge embedding (KE) models (e.g., TransE (Bordes et al., 2013)), and fixed during training PLMs. Thus they are not real joint models to learn the knowledge embedding and language embedding simultaneously. (2) The previous models only take entity embeddings to enhance PLMs, which are hard to fully capture the rich contextual information of an entity in the knowledge graph (KG). Thus their performance gains are limited by the quality of pre-trained entity embeddings. (3) The pre-trained entity embeddings are static and need to be re-trained when the KG is slightly changed. In this paper, we propose the Contextualized Language And Knowledge Embedding (CoLAKE), which jointly learns language representation and knowledge representation in a common representation space. Different from the previous models, CoLAKE dynamically represents an entity according to its knowledge context and language context. For each entity, CoLAKE considers a sub-graph surrounding ∗Work done during internship at Amazon …

Content truncated to 500 words