Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 416 - 431 May 22-27, 2022 c⃝2022 Association for Computational Linguistics So Different Yet So Alike! Constrained Unsupervised Text Style Transfer Abhinav Ramesh Kashyapα∗, Devamanyu Hazarikaα∗, Min-Yen Kanα, Roger Zimmermannα, Soujanya Poria DeCLaRe α National University of Singapore, Singapore DeCLaReDeCLaRe Lab, Singapore University of Technology and Design, Singapore {abhinav, hazarika, kanmy, rogerz}@comp.nus.edu.sg sporia@sutd.edu.sg Abstract Automatic transfer of text between domains has become popular in recent times. One of its aims is to preserve the semantic content of text being translated from source to target do- main. However, it does not explicitly main- tain other attributes between the source and translated text, for e.g., text length and descrip- tiveness. Maintaining constraints in transfer has several downstream applications, includ- ing data augmentation and de-biasing. We introduce a method for such constrained un- supervised text style transfer by introducing two complementary losses to the generative adversarial network (GAN) family of mod- els. Unlike the competing losses used in GANs, we introduce cooperative losses where the discriminator and the generator cooper- ate and reduce the same loss. The first is a contrastive loss and the second is a clas- sification loss — aiming to regularize the la- tent space further and bring similar sentences across domains closer together. We demon- strate that such training retains lexical, syntac- tic, and domain-specific constraints between domains for multiple benchmark datasets, in- cluding ones where more than one attribute change. We show that the complementary co- operative losses improve text quality, accord- ing to both automated and human evaluation measures. 1 1 Introduction Modern neural networks methods are capable of mapping data from one domain to another. Promi- nent examples include translation of text between languages (Vaswani et al., 2017; Artetxe et al., 2018; Lample et al., 2017), emoji creation from human faces (Taigman et al., 2017), and stylistic transfer of speech (Yuan et al., 2021). In Natu- ral Language Processing (NLP), the umbrella term attribute transfer (Jin et al., 2020b) (or domain ∗The first two authors contributed equally 1https://github.com/abhinavkashyap/dct I really loved Murakami’s book Personal Pronoun Proper Noun Loved the movie Text Style Transfer Smaller Length No Personal Pronoun No Proper Noun I absolutely enjoyed Spielberg’s direction Text Style Transfer + Constraints Similar Length Similar Personal Pronoun Domain appropriate Proper Noun vs. Figure 1: Illustrative example showing transfer of text from books to movies while maintaining constraints of identity. transfer) refers to similar methods2. The aim is to maximally preserve the semantics of the source sentence (“content”) but change other properties (“attributes”), such as sentiment (Jin et al., 2020b), expertise (Cao et al., 2020), formality (Rao and Tetreault, 2018) or a combination of them (Subra- manian et al., 2018). Text style transfer, a popular form of attribute transfer, regards “style” as any attribute that changes between datasets (Jin et al., 2020a). Build- ing on the progress of supervised transfer models, recent works have focused on unsupervised style transfer that avoids …

Content truncated to 500 words