Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4599–4610 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4599 Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement Bingzhi Li and Guillaume Wisniewski and Benoît Crabbé Université de Paris, LLF, CNRS 75 013 Paris, France bingzhi.li@etu.u-paris.fr {guillaume.wisniewski,benoit.crabbe}@u-paris.fr Abstract Many recent works have demonstrated that un- supervised sentence representations of neural networks encode syntactic information by ob- serving that neural language models are able to predict the agreement between a verb and its subject. We take a critical look at this line of research by showing that it is possi- ble to achieve high accuracy on this agreement task with simple surface heuristics, indicating a possible flaw in our assessment of neural net- works’ syntactic ability. Our fine-grained anal- yses of results on the long-range French object- verb agreement show that contrary to LSTMs, Transformers are able to capture a non-trivial amount of grammatical structure. 1 Introduction The long distance agreement task is one of the most popular method to assess neural networks (NN) ability to encode syntactic information: Linzen et al. (2016) showed that LSTMs are able to predict the subject-verb agreement in English and has initi- ated a very active line of research. Since then, many studies have generalized this observation to other languages (Gulordava et al., 2018), other models such as Transformers (Goldberg, 2019; Jawahar et al., 2019) or have identified possible confound- ing factors that could distort the stated conclusions (Gulordava et al., 2018; Marvin and Linzen, 2018). All of these studies show that NN are able to learn a ‘substantial amount’ of syntactic information (Be- linkov and Glass, 2019). In this work, we propose to take an alterna- tive look at these results by studying whether neu- ral networks are able to predict the correct form of a verb because they are able to build an ab- stract, high-level (maybe hierarchical) sentence representation (Giulianelli et al., 2018; Lakretz et al., 2019) or solely because they capture sur- face statistical regularities, as suggested by sev- eral recent work (Sennhauser and Berwick, 2018; Chaves, 2020; Li and Wisniewski, 2021). Overall, this set of results questions one of the most fun- damental assumption in linguistics (Lakretz et al., 2021), namely that a sentence has a recursive struc- ture (Everaert et al., 2015): while LSTMs with proper parametrization can model context-free pat- terns (Suzgun et al., 2019), Transformers are es- sentially feed forward models relying on a large number of attention heads. Consequently, they are, in theory, not adapted to model hierarchical syn- tactic patterns (Hahn, 2020) and explaining their capacity to predict accurately syntactic agreement patterns remains an open issue. We bring new light on this problematic by iden- tifying simple heuristics (§4) that can be used to correctly predict verbal agreement, pushing further the observation of Kuncoro et al. (2018) that a sim- ple rule can provide highly accurate results on the task. Using our extended set of heuristics, we …

Content truncated to 500 words