BERT Shows Garden Path Effects

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3220–3232 May 2-6, 2023 ©2023 Association for Computational Linguistics BERT Shows Garden Path Effects Tovah Irwin* and Kyra Wilson* Research Institute New York University Abu Dhabi {tovah.irwin, kyra.e.wilson22} @ gmail.com Alec Marantz Dept. of Psychology and Linguistics New York University marantz @ nyu.edu Abstract Garden path sentences (i.e. “the horse raced past the barn fell”) are sentences that read- ers initially incorrectly parse, requiring partial or total re-analysis of the sentence structure. Given human difficulty in parsing garden paths, we aim to compare transformer language mod- els’ performance on these sentences. We assess a selection of models from the BERT family which have been fine-tuned on the question- answering task, and evaluate each model’s per- formance on comprehension questions based on garden path and control sentences. We then further investigate the semantic roles assigned to arguments of verbs in garden path and con- trol sentences by utilizing a probe task to di- rectly assess which semantic role(s) the model assigns.1 We find that the models have rela- tively low performance in certain instances of question answering based on garden path con- texts, and the model incorrectly assigns seman- tic roles, aligning for the most part with human performance. 1 Introduction The field of natural language processing was revolu- tionized by the introduction of transformers. Mod- els such as BERT and GPT (and successors) have vastly improved performance on a variety of tasks compared to previous models such as LSTMs. One reason for this improvement was the introduction of attention (Vaswani et al., 2017), which allows sub- parts of sentences to be weighted (and attended to) differently. Another advance in these models was having all input processed simultaneously rather than sequentially. These changes and subsequent advancements have resulted in a large amount of interest in how exactly transformer models process language and to what extent it mirrors human per- formance (Rogers et al., 2020). 1Our code, datasets, and results are publicly available at https://github.com/kyrawilson/gardenBERT. In terms of syntactic information, investigations have found that BERT represents a variety of phe- nomena that are presumed to be relevant for human language processing such as parts of speech, roles, and syntactic chunks (Tenney et al., 2019; Liu et al., 2019a). Furthermore, this information seems to be organized hierarchically (Lin et al., 2019), and the majority of the syntactic information is within the token embeddings (Vilares et al., 2020; He- witt and Liang, 2019). Probing tasks have revealed that BERT contains semantic information as well. Roles, proto-roles, entity types, and relations are contained in token representations in addition to syntactic information (Ettinger, 2020; Tenney et al., 2019). Because much of the syntactic and semantic knowledge that humans are presumed to have and use is also present in BERT, it is potentially use- ful to compare the two in linguistic tasks and see if their performance is also similar. In this study, we compare the performance of humans and four …

Content truncated to 500 words