A Simple Geometric Method for Cross-Lingual Linguistic Transformations with Pre-trained Autoencoders

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10108–10114 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10108 A Simple Geometric Method for Cross-Lingual Linguistic Transformations with Pre-trained Autoencoders Maarten De Raedt Chatlayer.ai by Sinch Fréderic Godin Chatlayer.ai by Sinch Pieter Buteneers Sinch Chris Develder Ghent University - Imec Thomas Demeester Ghent University - Imec Abstract Powerful sentence encoders trained for multi- ple languages are on the rise. These systems are capable of embedding a wide range of lin- guistic properties into vector representations. While explicit probing tasks can be used to ver- ify the presence of specific linguistic proper- ties, it is unclear whether the vector represen- tations can be manipulated to indirectly steer such properties. For efficient learning, we in- vestigate the use of a geometric mapping in embedding space to transform linguistic prop- erties, without any tuning of the pre-trained sentence encoder or decoder. We validate our approach on three linguistic properties using a pre-trained multilingual autoencoder and ana- lyze the results in both monolingual and cross- lingual settings. 1 Introduction Recently, the design of sentence encoders, monolin- gual (Kiros et al., 2015; Conneau et al., 2017) and multilingual (Artetxe and Schwenk, 2019; Feng et al., 2020) has enjoyed a lot of attention. Many works have used probing tasks to investigate the presence of specific linguistic properties in sen- tence representations (Adi et al., 2016; Conneau et al., 2018; Conneau and Kiela, 2018; Ravishankar et al., 2019; Hewitt and Manning, 2019; Chi et al., 2020). However, it remains unclear to what extent these linguistic properties can be actually steered by manipulating the representations. By analogy to the definition of style-transfer from Li et al. (2018), we refer to modifying a particular linguistic prop- erty in a given text (e.g., a sentence’s tense) while preserving all of the property-independent content as linguistic property transfer. Training dedicated models to transfer linguistic properties requires substantial computational effort and a lot of training data. Adding the ability to transform a new property may require an entire retraining of the text encoder and decoder. This is especially challenging for low-resource languages or when reusing or building transfer models for more than one language. Assuming that pre-trained autoencoders capture the linguistic properties of interest, we investi- gate (i) whether they can be used without further tuning to efficiently transfer the properties, and (ii) whether this extends to the cross-lingual setting, when based on a multilingual pre-trained autoen- coder. Our starting point is a pre-trained sentence encoder, with a corresponding decoder trained on an autoencoder objective. We show how a geo- metric transformation of pre-trained multilingual sentence embeddings can be efficiently learned on CPU for transferring specific linguistic properties. We also experiment with cross-lingual linguistic property transfer, using a language-agnostic pre- trained encoder. In summary, this paper presents a set of prelim- inary experiments on linguistic property transfer, and shows that there may be value in further re- search on manipulating distributed representations to efficiently tackle language generation tasks. …

Content truncated to 500 words