BLM-AgrF: A New French Benchmark to Investigate Generalization of Agreement in Neural Networks

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1363–1374 May 2-6, 2023 ©2023 Association for Computational Linguistics BLM-AgrF: A New French Benchmark to Investigate Generalization of Agreement in Neural Networks Aixiu An1∗, Chunyang Jiang2, Maria A. Rodriguez3 *, Vivi Nastase2 and Paola Merlo2 1Department of Brain and Cognitive Sciences Massachusetts Institute of Technology, aixiu@mit.edu 2Department of Linguistics University of Geneva {Chunyang.Jiang, Paola.Merlo}@unige.ch, vivi.a.nastase@gmail.com 3 Computer Science and Information Technology Lucerne University of Applied Sciences and Arts, maria.anduezarodriguez@hslu.ch Abstract Successful machine learning systems currently rely on massive amounts of data, which are very effective in hiding some of the shallowness of the learned models. To help train models with more complex and compositional skills, we need challenging data, on which a system is suc- cessful only if it detects structure and regulari- ties, that will allow it to generalize. In this pa- per, we describe a French dataset (BLM-AgrF) for learning the underlying rules of subject-verb agreement in sentences, developed in the BLM framework, a new task inspired by visual IQ tests known as Raven’s Progressive Matrices. In this task, an instance consists of sequences of sentences with specific attributes. To predict the correct answer as the next element of the sequence, a model must correctly detect the generative model used to produce the dataset. We provide details and share a dataset built following this methodology. Two exploratory baselines based on commonly used architec- tures show that despite the simplicity of the phenomenon, it is a complex problem for deep learning systems. 1 Introduction Over the last years, driven by the surge in deep learning methods, models in NLP have become very powerful. They have even reached super- human performance on standard benchmarks such as SuperGLUE (Wang et al., 2019) and SQuAD (Rajpurkar et al., 2018). Deeper probing, though, shows that this is due to the models’ surprisingly robust superficial natural language understanding ∗The work was done while the author was at the Univer- sity of Geneva. Figure 1: An example Raven’s progressive matrix (best seen in colour). The matrix is constructed according to two rules: (i) the red dot moves one place clockwise when traversing the matrix left to right; (ii) the blue square moves one place anticlockwise when traversing the matrix top to bottom. The task consists in finding the tile in the answer set that correctly completes the sequence, indicated with a double border. ability. This indicates that new benchmarks are nec- essary that accurately show the level of progress in language processing (Ruder, 2021). In this paper, we describe such a benchmark, developed based on a new method described in (Merlo et al., 2022), and summarized in Section 2. The method defines a procedure for building datasets that capture specific linguistic phenom- ena in a structured problem, inspired by the visual patterns detection tasks in Raven’s progressive ma- trices (RPM) (Raven, 1938), as in Figure 1. The visual RPMs manipulate elements with at- tributes such as position, shape, colour …

Content truncated to 500 words