Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7181–7198 July 9-14, 2023 ©2023 Association for Computational Linguistics Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features Ester Hlavnova Sebastian Ruder Google Research {ehlavnova,ruder}@google.com Abstract A challenge towards developing NLP systems for the world’s languages is understanding how they generalize to typological differences relevant for real-world applications. To this end, we propose M2C, a morphologically- aware framework for behavioral testing of NLP models. We use M2C to generate tests that probe models’ behavior in light of spe- cific linguistic features in 12 typologically di- verse languages. We evaluate state-of-the-art language models on the generated tests. While models excel at most tests in English, we high- light generalization failures to specific typo- logical characteristics such as temporal expres- sions in Swahili and compounding possessives in Finish. Our findings motivate the develop- ment of models that address these blind spots.1 1 Introduction In natural language processing (NLP), there is a need to build systems that serve more of the world’s approximately 6,900 languages. As one measure of linguistic diversity, the World Atlas of Lan- guage Structures (WALS; Haspelmath et al., 2005) records 192 linguistic features along which lan- guages differ. These range from the order of sub- ject, object, and verb (Dryer, 2013) to the number of basic color categories (Kay and Maffi, 2013). Languages present in existing NLP datasets mostly lie in low-density regions of the space of possible typological features (Ponti et al., 2021). In other words, many linguistic features that are common across the world’s languages are not observed in languages that are the focus of NLP research.2 It is thus important to investigate to which lin- guistic features models can generalize and where they face challenges. However, existing datasets 1We make all code publicly available at https://github. com/google-research/multi-morph-checklist. 2For instance, while tone is present in around 80% of African languages (Adebara and Abdul-Mageed, 2022), few Indo-European languages can be considered tonal. 0.0 25.0 50.0 75.0 100.0 English Slovak Chinese Swahili Average (across 12 languages) mT5-XXL PaLM-S PaLM-M PaLM-L PaLM 2 Figure 1: Top: Comparison of state-of-the-art models on M2C tests in a selected set of languages. Models perform well on English but poorly on certain tests in other languages. Bottom: Even the largest models fail on tests probing language-specific features, e.g., the dis- tinction between habitual and one-time motion verbs in Russian (left) or possessives in Finnish (right); see Ap- pendix B for English glosses and additional examples. do not allow for a fine-grained cross-lingual evalua- tion and mainly permit comparisons on a language level (Hu et al., 2020). Prior studies focused on syn- tax and grammar through the lens of acceptability judgements (Ravfogel et al., 2018; Ahmad et al., 2019; Mueller et al., 2020; Papadimitriou et al., 2022). While these enable the evaluation of what a model deems ‘natural’ in a given language, it is often unclear how such biases relate to real-world applications of NLP technology. We …

Content truncated to 500 words