Contextualized Embeddings for Enriching Linguistic Analyses on Politeness

Proceedings of the 28th International Conference on Computational Linguistics, pages 2181–2190 Barcelona, Spain (Online), December 8-13, 2020 2181 Contextualized Embeddings for Enriching Linguistic Analyses on Politeness Ahmad Aljanaideh and Eric Fosler-Lussier Department of Computer Science and Engineering The Ohio State University Marie-Catherine de Marneffe Department of Linguistics The Ohio State University Abstract Linguistic analyses in natural language processing (NLP) have often been performed around the static notion of words where the context (surrounding words) is not considered. For example, previous analyses on politeness have focused on comparing the use of static words such as personal pronouns across (im)polite requests without taking the context of those words into account. Current word embeddings in NLP do capture context and thus can be leveraged to enrich linguistic analyses. In this work, we introduce a model which leverages the pre-trained BERT model to cluster contextualized representations of a word based on (1) the context in which the word appears and (2) the labels of items the word occurs in. Using politeness as case study, this model is able to automatically discover interpretable, fine-grained context patterns of words, some of which align with existing theories on politeness. Our model further discovers novel finer-grained patterns associated with (im)polite language. For example, the word please can occur in impolite contexts that are predictable from BERT clustering. The approach proposed here is validated by showing that features based on fine-grained patterns inferred from the clustering improve over politeness-word baselines. 1 Introduction Linguistic analyses have often been computationally performed around the static notion of words or word categorization methods (e.g. LIWC) where the context of words is not taken into account. Previous work on politeness (Danescu-Niculescu-Mizil et al., 2013) and gender (Bamman et al., 2014) have focused on comparing the use of non-contextual broad word categories such as personal pronouns across different categories/groups. For example, Bamman et al. (2014) found that women use more pronoun words than men. Danescu-Niculescu-Mizil et al. (2013) found that requests which contain a hedge word (e.g. think) are more likely to be perceived as polite than impolite. However, words often occur in many different contexts and thus analyzing them statically hides cues which can potentially enrich our understanding of the phenomenon being studied. As opposed to static word embeddings which provide the same representation for a word regardless of its context (i.a. Mikolov et al., 2013a; Mikolov et al., 2013b; Pennington et al., 2014), the BERT (Devlin et al., 2018) and ELMo (Peters et al., 2018) models provide methods for extracting pre-trained contextualized word representations. By leveraging contextualized representations, linguistic theories can be validated and enriched. Given a dataset annotated for a downstream task, we build a model which automatically discovers fine-grained context patterns of words. Discovering such patterns provides insight into the phenomenon the task attempts to model. We use pre-trained BERT embeddings and exploit the fact that words which occur in similar contexts tend to have similar representations. Our model takes as input contextualized representations for a given word and splits them …

Content truncated to 500 words