Using Paraphrases to Study Properties of Contextual Embeddings

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4558 - 4568 July 10-15, 2022 ©2022 Association for Computational Linguistics Using Paraphrases to Study Properties of Contextual Embeddings Laura Burdick, Jonathan K. Kummerfeld and Rada Mihalcea Computer Science & Engineering University of Michigan, Ann Arbor {lburdick,jkummerf,mihalcea}@umich.edu Abstract We use paraphrases as a unique source of data to analyze contextualized embeddings, with a particular focus on BERT. Because para- phrases naturally encode consistent word and phrase semantics, they provide a unique lens for investigating properties of embeddings. Using the Paraphrase Database’s alignments, we study words within paraphrases as well as phrase representations. We find that con- textual embeddings effectively handle polyse- mous words, but give synonyms surprisingly different representations in many cases. We confirm previous findings that BERT is sensi- tive to word order, but find slightly different patterns than prior work in terms of the level of contextualization across BERT’s layers. 1 Introduction Contextualized embedding algorithms, such as BERT (Devlin et al., 2019), have achieved impres- sive performance on a wide variety of tasks (Huang et al., 2019; Chan and Fan, 2019; Yoosuf and Yang, 2019). One application of BERT is using it as a measure of sentence similarity (Zhang* et al., 2020; Sellam et al., 2020), based on the assumption that BERT will produce similar representations for the words in two sentences with similar semantics. We propose to use paraphrases with alignments between words as a tool for studying how BERT represents words and phrases. Figure 1 shows an example. Critically, when considering an aligned word pair, we can assume the context has a similar impact on both words because we know the phrases are semantically similar. Previously, paraphrases have been used to probe whether compositionality is accurately captured by BERT (Yu and Ettinger, 2020), but we believe they can be used to explore many other questions. Using the second version of the Paraphrase Figure 1: Example paraphrase from the PPDB with word alignment and word cosine similarities using the last layer of BERT. Database (PPDB, Pavlick et al., 2015), we ex- plore how consistent contextual representations are when controlling for the semantics of the con- text. First, we use the human-annotated portion of the PPDB data to confirm that BERT consis- tently represents paraphrases. Next, we use the highest-quality (but not all human-annotated) sec- tion of the PPDB to probe BERT’s behavior in more detail. Looking at words, BERT effectively handles variations in spelling, but does less well with spelling errors. BERT also effectively han- dles words of varying levels of polysemy, but the representations for synonyms are surprisingly di- verse, with a much broader distribution of similarity scores. These findings confirm results from prior work using other methods, while uncovering new insights about contextual embedding models. We also consider a range of other models’ word representations, finding that they have similar pat- terns to BERT, but with aligned words that are the same receiving even more …

Content truncated to 500 words