How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 108–118 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 108 How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation Steffen Eger, Johannes Daxenberger, Iryna Gurevych Computer Science Department, Technische Universit¨at Darmstadt, Germany eger@aiphes.tu-darmstadt.de, {daxenberger,gurevych}@ukp.informatik.tu-darmstadt.de Abstract Sentence encoders map sentences to real val- ued vectors for use in downstream applica- tions. To peek into these representations—e.g., to increase interpretability of their results— probing tasks have been designed which query them for linguistic knowledge. However, de- signing probing tasks for lesser-resourced lan- guages is tricky, because these often lack large- scale annotated data or (high-quality) depen- dency parsers as a prerequisite of probing task design in English. To investigate how to probe sentence embeddings in such cases, we in- vestigate sensitivity of probing task results to structural design choices, conducting the first such large scale study. We show that de- sign choices like size of the annotated prob- ing dataset and type of classifier used for eval- uation do (sometimes substantially) influence probing outcomes. We then probe embeddings in a multilingual setup with design choices that lie in a ‘stable region’, as we identify for En- glish, and find that results on English do not transfer to other languages. Fairer and more comprehensive sentence-level probing evalua- tion should thus be carried out on multiple lan- guages in the future. 1 Introduction Sentence embeddings (a.k.a. sentence encoders) have become ubiquitous in NLP (Kiros et al., 2015; Conneau et al., 2017), extending the concept of word embeddings to the sentence level. In the con- text of recent efforts to open the black box of deep learning models and representations (Linzen et al., 2019), it has also become fashionable to probe sentence embeddings for the linguistic informa- tion signals they contain (Perone et al., 2018), as this may not be clear from their performances in downstream tasks. Such probes are linguistic micro tasks—like detecting the length of a sentence or its dependency tree depth—that have to be solved by classifier LR MLP NB RF size High (A,B,C) (A,B,C) (C,A,B) (C,B,A) Mid (A,C,B) (C,B,A) (A,B,C) (C,B,A) Low (A,B,C) (B,A,C) (B,C,A) (A,B,C) Table 1: Schematic illustration of our concept of stabil- ity across two dimensions (classifier and training size). Here, three encoders, dubbed A,B,C, are ranked. The region of stability is given by those settings that support the majority ranking of encoders, which is A≻B≻C. a classifier using given representations. The majority of approaches for probing sen- tence embeddings target English, but recently some works have also addressed other languages such as Polish, Russian, or Spanish in a multi- and cross-lingual setup (Krasnowska-Kiera´s and Wr´oblewska, 2019; Ravishankar et al., 2019). Mo- tivations for considering a multi-lingual analysis in- clude knowing whether findings from English trans- fer to other languages and determining a universal set of probing tasks that suits multiple languages, e.g., with richer morphology and freer word order. Our work is also inspired by probing sentence …

Content truncated to 500 words