DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2540–2549 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2540 DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections Yury Zemlyanskiy∗ USC zemlyans@usc.edu Sudeep Gandhe Google Research srgandhe@google.com Ruining He Google Research ruininghe@google.com Bhargav Kanagal Google Research bhargav@google.com Anirudh Ravula Google Research braineater@google.com Juraj Gottweis Google Research juro@google.com Fei Sha† Google Research fsha@google.com Ilya Eckstein Google Research ilyaeck@google.com Abstract This paper explores learning rich self- supervised entity representations from large amounts of associated text. Once pre-trained, these models become applicable to multiple entity-centric tasks such as ranked retrieval, knowledge base completion, question an- swering, and more. Unlike other methods that harvest self-supervision signals based merely on a local context within a sentence, we radically expand the notion of context to include any available text related to an entity. This enables a new class of powerful, high- capacity representations that can ultimately distill much of the useful information about an entity from multiple text sources, without any human supervision. We present several training strategies that, un- like prior approaches, learn to jointly predict words and entities—strategies we compare ex- perimentally on downstream tasks in the TV- Movies domain, such as MovieLens tag pre- diction from user reviews and natural language movie search. As evidenced by results, our models match or outperform competitive base- lines, sometimes with little or no fine-tuning, and can scale to very large corpora. Finally, we make our datasets and pre-trained models publicly available1. This includes Reviews2Movielens, mapping the ∼1B word corpus of Amazon movie reviews (He and McAuley, 2016) to MovieLens tags (Harper and Konstan, 2016), as well as Reddit Movie Suggestions with natural language queries and corresponding community recommendations. 1 Introduction Much of the online information describing enti- ties in domains such as music, movies, venues or ∗Work is partially done while at Google †On leave from USC (feisha@usc.edu) 1See http://goo.gle/research-docent for Re- views2Movielens and models. Scripts and Reddit Suggestions can be found at https://urikz.github.io/docent Review 1: “This movie develops its power best if you don’t try to look out for the “real” and “true” events behind the four versions of the narration... shown in a very intelligent and artistic way, no silly plot-twists, no explanation in the end — it is open to your fantasy... “$MOVIE” is an important piece of cinematic storytelling and a really interesting way to reflect on the origin of tales... Some scenes even remind me of Andrej Tarkovskijs intensive style..”. Review 2: “Just rented this, and at first I didn’t like very much, but then it starts to sink in for how good it is, the acting is great especially Toshiro Mifune, it was shot very good for an older movie... it’s #62 on the top 250” Review 3: “Saw this movie at my local video store... was placed on a waiting list, but when I returned to check it out the video store had closed down over night. Actually whent out …

Content truncated to 500 words