Speaker Information Can Guide Models to Better Inductive Biases: A Case Study On Predicting Code-Switching

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3853 - 3867 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Speaker Information Can Guide Models to Better Inductive Biases: A Case Study On Predicting Code-Switching Alissa Ostapenko1 Shuly Wintner2 Melinda Fricke3 Yulia Tsvetkov4 1Language Technologies Institute, Carnegie Mellon University 2Department of Computer Science, University of Haifa 3Department of Linguistics, University of Pittsburgh 4Paul G. Allen School of Computer Science & Engineering, University of Washington aostapen@andrew.cmu.edu, shuly@cs.haifa.ac.il, melinda.fricke@pitt.edu, yuliats@cs.washington.edu Abstract Natural language processing (NLP) models trained on people-generated data can be un- reliable because, without any constraints, they can learn from spurious correlations that are not relevant to the task. We hypothesize that enrich- ing models with speaker information in a con- trolled, educated way can guide them to pick up on relevant inductive biases. For the speaker- driven task of predicting code-switching points in English–Spanish bilingual dialogues, we show that adding sociolinguistically-grounded speaker features as prepended prompts signif- icantly improves accuracy. We find that by adding influential phrases to the input, speaker- informed models learn useful and explainable linguistic information. To our knowledge, we are the first to incorporate speaker characteris- tics in a neural model for code-switching, and more generally, take a step towards develop- ing transparent, personalized models that use speaker information in a controlled way. 1 Introduction Imbalanced datasets, flawed annotation schemes, and even model architectures themselves can all cause neural models to encode and propagate bi- ases by incorrectly correlating social information with labels for a task (Sun et al., 2019; Field et al., 2021). As a result, models may be brittle and offen- sive in the presence of racial or gender attributes (Kiritchenko and Mohammad, 2018; Nozza et al., 2021), unsuitable for processing mixed-language text or dialect variations (Sap et al., 2019; Kumar et al., 2021; Winata et al., 2021), or ones that can miscommunicate intents in translation setups. Con- textualizing models in social factors is important for preventing these issues and building more so- cially intelligent and culturally sensitive NLP tech- nologies (Hovy and Yang, 2021). We hypothesize that grounding models in speaker information can help them learn more useful inductive biases, thereby improving per- formance on person-oriented classification tasks. We test this hypothesis on the task of predicting code-switching (language mixing) in a multilin- gual dialogue, which is inherently linguistically and socially driven (Li, 2013). Prior approaches for predicting code-switching consider only shal- low linguistic context (Do˘gruöz et al., 2021). As we show in our experiments1, using a standard Transformer-based classifier (Conneau et al., 2020) trained with only linguistic context results in sub- optimal and unstable models. Moreover, we believe code-switch prediction is a suitable first task for learning speaker-driven inductive biases; we can test whether models learn useful relationships be- tween social attributes while minimizing the risk of building a model that perpetuates social prejudices. We ground the models in relevant social fac- tors, such as age, native language, and language- mixing preference of …

Content truncated to 500 words