Probing for the Usage of Grammatical Number
37094Authors
nan
Abstract
AbstractA central quest of probing is to uncover how pre-trained models encode a linguistic property within their representations. An encoding, however, might be spurious—i.e., the model might not rely on it when making predictions. In this paper, we try to find an encoding that the model actually uses, introducing a usage-based probing setup. We first choose a behavioral task which cannot be solved without using the linguistic property. Then, we attempt to remove the property by intervening on the model’s representations. We contend that, if an encoding is used by the model, its removal should harm the performance on the chosen behavioral task. As a case study, we focus on how BERT encodes grammatical number, and on how it uses this encoding to solve the number agreement task. Experimentally, we find that BERT relies on a linear encoding of grammatical number to produce the correct behavioral output. We also find that BERT uses a separate encoding of grammatical number for nouns and verbs. Finally, we identify in which layers information about grammatical number is transferred from a noun to its head verb.
Keywords
Full Content
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 70 - 80 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Domain Adaptation in Multilingual and Multi-Domain Monolingual Settings for Complex Word Identification George-Eduard Zaharia, R˘azvan-Alexandru Sm˘adu Dumitru-Clementin Cercel, Mihai Dascalu University Politehnica of Bucharest, Faculty of Automatic Control and Computers {george.zaharia0806, razvan.smadu}@stud.acs.upb.ro {dumitru.cercel, mihai.dascalu}@upb.ro Abstract Complex word identification (CWI) is a cor- nerstone process towards proper text simpli- fication. CWI is highly dependent on con- text, whereas its difficulty is augmented by the scarcity of available datasets which vary greatly in terms of domains and languages. As such, it becomes increasingly more difficult to develop a robust model that generalizes across a wide array of input examples. In this paper, we propose a novel training technique for the CWI task based on domain adaptation to im- prove the target character and context represen- tations. This technique addresses the problem of working with multiple domains, inasmuch as it creates a way of smoothing the differences between the explored datasets. Moreover, we also propose a similar auxiliary task, namely text simplification, that can be used to com- plement lexical complexity prediction. Our model obtains a boost of up to 2.42% in terms of Pearson Correlation Coefficients in contrast to vanilla training techniques, when consider- ing the CompLex from the Lexical Complex- ity Prediction 2021 dataset. At the same time, we obtain an increase of 3% in Pearson scores, while considering a cross-lingual setup rely- ing on the Complex Word Identification 2018 dataset. In addition, our model yields state-of- the-art results in terms of Mean Absolute Er- ror. 1 Introduction The overarching goal of the complex word iden- tification (CWI) task is to find words that can be simplified in a given text (Paetzold and Specia, 2016b). Evaluating word difficulty represents one step towards achieving simplified, which in return facilitates access to knowledge to a wider audience texts (Maddela and Xu, 2018). However, complex word identification is a highly contextualized task, far from being trivial. The datasets are scarce and, most of the time, the input entries are limited or cover different domains/areas of expertise. There- Domain Text CompLex LCP Dataset Bible But let each man test his own work , and then he will take pride in himself and not in his neighbor. Biomedical A genome database search revealed orthologs of ADAM11, ADAM22 and ADAM23 genes to exist in vertebrates such as mammals, fish, and amphibians, but not in invertebrates. Europarl They also allow for easy compensation for the thou- sands of accidents involving vehicles from more than one Member State. English CWI Dataset Wikipedia Normally , the land will be passed down to future generations in a way that recognizes the community’s traditional connection to that country . WikiNews The JAS 39C Gripen crashed onto a runway at around 9:30 am local time (02:30 UTC) and exploded , closing the airport to commercial flights. News The car has been removed from the scene …
Content truncated to 500 words
Labeled Text
AbstractA central quest of probing is to uncover how pre-trained models encode a linguistic property within their representations. An encoding, however, might be spurious—i.e., the model might not rely on it when making predictions. In this paper, we try to find an encoding that the model actually uses, introducing a usage-based probing setup. We first choose a behavioral task which cannot be solved without using the linguistic property. Then, we attempt to remove the property by intervening on the model’s representations. We contend that, if an encoding is used by the model, its removal should harm the performance on the chosen behavioral task. As a case study, we focus on how BERT encodes grammatical number, and on how it uses this encoding to solve the number agreement task. Experimentally, we find that BERT relies on a linear encoding of grammatical number to produce the correct behavioral output. We also find that BERT uses a separate encoding of grammatical number for nouns and verbs. Finally, we identify in which layers information about grammatical number is transferred from a noun to its head verb.
Publication Details
- Paper ID:
- 37094
- Year:
- 2022
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan