Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4875 - 4893 July 10-15, 2022 ©2022 Association for Computational Linguistics Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks Ruixiang Cui, Daniel Hershcovich, Anders Søgaard University of Copenhagen {rc, dh, soegaard}@di.ku.dk Abstract Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today’s NLU models still strug- gle to capture their semantics. We rely on Generalized Quantifier Theory for language- independent representations of the semantics of quantifier words, to quantify their contribu- tion to the errors of NLU models. We find that quantifiers are pervasive in NLU bench- marks, and their occurrence at test time is as- sociated with performance drops. Multilingual models also exhibit unsatisfying quantifier rea- soning abilities, but not necessarily worse for non-English languages. To facilitate directly- targeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quanti- fier reasoning. 1 Introduction Quantifier words—such as each or most or more than three—have been extensively studied, both in logic and in linguistics (Westerståhl, 1989; Peters and Westerståhl, 2006), going all the way back to Frege (1879). In this paper, we examine the extent to which they present a challenge to modern NLU systems. Our analysis is motivated by three observations: Quantifier words are abstract Unlike nouns, verbs and adjectives, quantifier words do not have referents out in the world. Rather, quantifier words specify relationships between sets of entities, events and properties. To provide intuitions about the semantics of quantifier words, and to be able to refer to quantifiers in a language-independent way, we rely on the notion of generalized quantifiers (Mostowski, 1957), as described in §2. Quantifier words vary across languages Quantifier word inventories differ across languages. QA_English CONTEXT: A piece of paper was later found on which he had written his last statements in two languages, Latin and German. Only one statement was in Latin and the rest in German. QUESTION: In what language were most statements written? ANSWER: German PREDICTED AN- SWER: Latin and German NLI_Spanish PREMISE: Más de tres personas resultaron heridas en un accidente de dos vehículos el lunes por la noche. (translation: More than three people were injured in a two-vehicle crash Monday evening.) HYPOTHESIS: Había 4 personas involucradas. (trans- lation: There were 4 people involved. LABEL: Neutral PREDICTED LABEL: Entailment Table 1: Examples of quantifiers (marked in bold texts) in NLP tasks, with RoBERTa’s prediction for QA and XLM-R’s prediction for NLI after fine-tuning. Often what is considered rough translation equiva- lents also differ in syntax, fine-grained semantics or pragmatics. Stateva et al. (2019) show, e.g., that perceptions of the numerical bounds of ex- istential quantifiers differ across speakers of En- glish, French, Slovenian, and German. Other pa- pers showing discrepancies between quantifier sys- tems include comparisons of Salish to English (Matthewson, 2001), Adyghe to …

Content truncated to 500 words