Can Edge Probing Tests Reveal Linguistic Knowledge in QA Models?
5078Authors
nan
Abstract
AbstractThere have been many efforts to try to understand what grammatical knowledge (e.g., ability to understand the part of speech of a token) is encoded in large pre-trained language models (LM). This is done through ‘Edge Probing’ (EP) tests: supervised classification tasks to predict the grammatical properties of a span (whether it has a particular part of speech) using only the token representations coming from the LM encoder. However, most NLP applications fine-tune these LM encoders for specific tasks. Here, we ask: if an LM is fine-tuned, does the encoding of linguistic information in it change, as measured by EP tests? Specifically, we focus on the task of Question Answering (QA) and conduct experiments on multiple datasets. We find that EP test results do not change significantly when the fine-tuned model performs well or in adversarial situations where the model is forced to learn wrong correlations. From a similar finding, some recent papers conclude that fine-tuning does not change linguistic knowledge in encoders but they do not provide an explanation. We find that EP models are susceptible to exploiting spurious correlations in the EP datasets. When this dataset bias is corrected, we do see an improvement in the EP test results as expected.
Keywords
Full Content
Proceedings of the 29th International Conference on Computational Linguistics, pages 141–151 October 12–17, 2022. 141 Revisiting Statistical Laws of Semantic Shift in Romance Cognates Yoshifumi Kawasaki The University of Tokyo ykawasaki@g.ecc.u-tokyo.ac.jp Maëlys Salingre Mejiro University m.salingre@mejiro.ac.jp Marzena Karpinska University of Massachusetts Amherst mkarpinska@cs.umass.edu Hiroya Takamura AIST takamura.hiroya@aist.go.jp Ryo Nagata Konan University nagata-coling2022@ml.hyogo-u.ac.jp Abstract This article revisits statistical relationships across Romance cognates between lexical se- mantic shift and six intra-linguistic variables, such as frequency and polysemy. Cognates are words that are derived from a common etymon, in this case, a Latin ancestor. Despite their shared etymology, some cognate pairs have ex- perienced semantic shift. The degree of se- mantic shift is quantified using cosine distance between the cognates’ corresponding word em- beddings. In the previous literature, frequency and polysemy have been reported to be cor- related with semantic shift; however, the un- derstanding of their effects needs revision be- cause of various methodological defects. In the present study, we perform regression anal- ysis under improved experimental conditions, and demonstrate a genuine negative effect of frequency and positive effect of polysemy on semantic shift. Furthermore, we reveal that morphologically complex etyma are more re- sistant to semantic shift and that the cognates that have been in use over a longer timespan are prone to greater shift in meaning. These find- ings add to our understanding of the historical process of semantic change. 1 Introduction The Romance languages, such as present-day French, Italian, and Spanish, are sister languages which evolved from Vulgar Latin (Alkire and Rosen, 2010). They share numerous cognates, that is, words derived from a common etymon: for in- stance, the Latin verb HABERE “to have” developed into avoir (fr), avere (it), and haber (es).1 Whereas French and Italian still maintain the original mean- ing “to have”, it is no longer used as such in Span- ish, where it is primarily used as an auxiliary to 1Hereinafter, “fr” stands for French, “it” for Italian, and “es” for Spanish. Latin words are put in small capitals. form perfect tense, while the notion of possession is generally expressed with tener. Likewise, the meanings and functions of some cognate pairs have diverged despite their common etymology. In the current paper, we investigate factors at play in se- mantic shift2 using a computational approach. In recent years, the analysis of diachronic se- mantic change using computational methods, inter alia, distributed representation of words, has gained increasing research interest (Dubossarsky et al., 2016; Hamilton et al., 2016; Takamura et al., 2017; Kutuzov et al., 2018; Uban et al., 2019; Hengchen et al., 2021; Kutuzov et al., 2021; Montariol and Al- lauzen, 2021; Schlechtweg et al., 2021; Tahmasebi et al., 2021; Uban et al., 2021a). The degree of lex- ical semantic shift has been conventionally quanti- fied using cosine distance between relevant words, represented in the form of embeddings. A num- ber of researchers have investigated the relation- ships between semantic change and frequency, pol- ysemy, or prototypicality (Dubossarsky et al., 2015; Hamilton et al., 2016; Uban …
Content truncated to 500 words
Labeled Text
AbstractThere have been many efforts to try to understand what grammatical knowledge (e.g., ability to understand the part of speech of a token) is encoded in large pre-trained language models (LM). This is done through ‘Edge Probing’ (EP) tests: supervised classification tasks to predict the grammatical properties of a span (whether it has a particular part of speech) using only the token representations coming from the LM encoder. However, most NLP applications fine-tune these LM encoders for specific tasks. Here, we ask: if an LM is fine-tuned, does the encoding of linguistic information in it change, as measured by EP tests? Specifically, we focus on the task of Question Answering (QA) and conduct experiments on multiple datasets. We find that EP test results do not change significantly when the fine-tuned model performs well or in adversarial situations where the model is forced to learn wrong correlations. From a similar finding, some recent papers conclude that fine-tuning does not change linguistic knowledge in encoders but they do not provide an explanation. We find that EP models are susceptible to exploiting spurious correlations in the EP datasets. When this dataset bias is corrected, we do see an improvement in the EP test results as expected.
Publication Details
- Paper ID:
- 5078
- Year:
- 2022
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan