Empirical Linguistic Study of Sentence Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5729–5739 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5729 Empirical Linguistic Study of Sentence Embeddings Katarzyna Krasnowska-Kiera´s Alina Wróblewska Institute of Computer Science, Polish Academy of Sciences ul. Jana Kazimierza 5, 01-248 Warsaw, Poland kasia.krasnowska@gmail.com alina@ipipan.waw.pl Abstract The purpose of the research is to answer the question whether linguistic information is retained in vector representations of sentences. We introduce a method of analysing the con- tent of sentence embeddings based on univer- sal probing tasks, along with the classification datasets for two contrasting languages. We per- form a series of probing and downstream ex- periments with different types of sentence em- beddings, followed by a thorough analysis of the experimental results. Aside from depen- dency parser-based embeddings, linguistic in- formation is retained best in the recently pro- posed LASER sentence embeddings. 1 Introduction Modelling natural language with neural networks has been an extensively researched area for sev- eral years now. On the one hand, deep learning enormously reduced the cost of feature engineer- ing. On the other hand, we are largely unaware of features that are used in estimating a neural model and, therefore, kinds of information that a trained neural model relies most heavily on. Since neu- ral network-based models work very well in many NLP tasks and often provide state-of-the-art re- sults, it is extremely interesting and desirable to understand which properties of words, phrases or sentences are retained in their embeddings. An ap- proach to investigate whether linguistic properties of English sentences are encoded in their embed- dings is proposed by Shi et al. (2016), Adi et al. (2017), and Conneau et al. (2018). It consists in designing a series of classification problems focus- ing on linguistic properties of sentences, so called probing tasks (Conneau et al., 2018). In a probing task, sentences are labelled according to a particu- lar linguistic property. Given a model that generates an embedding vector for any sentence, the model is applied to the probing sentences. A classifier is then trained with the resulting embeddings as in- puts and probing labels as targets. The performance of the resulting classifier is considered a proxy for how well the probing property is retained in the sen- tence embeddings. We propose an extension and generalisation of the methodology of the probing tasks-based experi- ments. First, the current experiments are conducted on two typologically and genetically different lan- guages: English, which is an isolating Germanic language and Polish, which is a fusional Slavic one. Our motivation for conducting experiments on two contrasting languages is as follows. English is undoubtedly the most prominent language with multiple resources and tools. However, English lan- guage processing is only a part of NLP in general. Methods designed for English are not guaranteed to be universal. In order to verify whether an NLP algorithm is powerful, it is not enough to evaluate it solely on English. Evaluation on additional lan- guages can …

Content truncated to 500 words