Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10837–10851 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10837 Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration Shufan Wang and Laure Thompson and Mohit Iyyer University of Massachusetts, Amherst {shufanwang, laurejt, miyyer}@cs.umass.edu Abstract Phrase representations derived from BERT of- ten do not exhibit complex phrasal composi- tionality, as the model relies instead on lex- ical similarity to determine semantic related- ness. In this paper, we propose a contrastive fine-tuning objective that enables BERT to pro- duce more powerful phrase embeddings. Our approach (Phrase-BERT) relies on a dataset of diverse phrasal paraphrases, which is auto- matically generated using a paraphrase gener- ation model, as well as a large-scale dataset of phrases in context mined from the Books3 corpus. Phrase-BERT outperforms baselines across a variety of phrase-level similarity tasks, while also demonstrating increased lexical di- versity between nearest neighbors in the vec- tor space. Finally, as a case study, we show that Phrase-BERT embeddings can be easily integrated with a simple autoencoder to build a phrase-based neural topic model that inter- prets topics as mixtures of words and phrases by performing a nearest neighbor search in the embedding space. Crowdsourced evalua- tions demonstrate that this phrase-based topic model produces more coherent and meaning- ful topics than baseline word and phrase-level topic models, further validating the utility of Phrase-BERT. 1 Introduction Learning representations of phrases is important for many tasks, such as semantic parsing (Socher et al., 2011), machine translation (Ramisch et al., 2013), and question answering (Seo et al., 2018). While pretrained language models such as BERT (Devlin et al., 2018) have significantly pushed for- ward the state of the art on a variety of NLP tasks, they still struggle to produce semantically mean- ingful embeddings for shorter linguistic units such as sentences and phrases. An out-of-the-box BERT sentence embedding model often underperforms simple baselines such as averaging GloVe vectors in semantic textual similarity tasks (Reimers and Model Nearest neighbors of “pulls the trigger" GloVe his trigger, the trigger, a trigger BERT pulled the trigger, squeezed the trigger, scoots closer Span- BERT pulled the trigger, pulling the trigger, seize the day Sent- BERT pulling the trigger, pulled the trigger, the trigger Phrase- BERT picks up his gun, squeezes off a quick burst of shots, takes aim Table 1: Nearest neighbors of the phrase “pulls the trig- ger”. While baselines rely heavily on lexical overlap, Phrase-BERT’s neighbors are both semantically simi- lar and lexically diverse. Gurevych, 2019). Furthermore, Yu and Ettinger (2020) have shown that phrasal representations de- rived from BERT do not exhibit complex phrasal compositionality. In this paper, we develop Phrase-BERT, which fine-tunes BERT using contrastive learning to in- duce more powerful phrase embeddings. Our approach directly targets two major weaknesses of out-of-the-box BERT phrase embeddings: (1) BERT never sees short texts (e.g., phrases) dur- ing pretraining, as its inputs are chunks of 512 tokens; and (2) BERT relies heavily on lexical sim- …

Content truncated to 500 words