Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4937 - 4948 December 7-11, 2022 ©2022 Association for Computational Linguistics Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables Erxin Yu1,2, Lan Du4, Yuan Jin4, Zhepei Wei1,2, Yi Chang1,2,3∗ 1School of Artificial Intelligence, Jilin University 2Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University 3International Center of Future Science, Jilin University 4Faculty of Information Technology, Monash University, Australia erxin.yu@outlook.com, lan.du@monash.edu, yuan.jin@monash.edu, weizp19@mails.jlu.edu.cn, yichang@jlu.edu.cn Abstract Recently, discrete latent variable models have received a surge of interest in both Natural Language Processing (NLP) and Computer Vision (CV), attributed to their comparable performance to the continuous counterparts in representation learning, while being more interpretable in their predictions. In this pa- per, we develop a topic-informed discrete la- tent variable model for semantic textual sim- ilarity, which learns a shared latent space for sentence-pair representation via vector quanti- zation. Compared with previous models lim- ited to local semantic contexts, our model can explore richer semantic information via topic modeling. We further boost the performance of semantic similarity by injecting the quan- tized representation into a transformer-based language model with a well-designed semantic- driven attention mechanism. We demonstrate, through extensive experiments across various English language datasets, that our model is able to surpass several strong neural baselines in semantic textual similarity tasks. 1 Introduction Semantic Textual Similarity (STS), which concerns the problem of measuring and scoring the relation- ships or relevance of pairs of text on real-valued scales, is a fundamental task in NLP. It has further driven many other important NLP tasks such as machine translation (Zou et al., 2013), text summa- rization (Mohamed and Oussalah, 2019), question answering (Bordes et al., 2014), and etc. Recently, deep neural language models have achieved state-of-the-art performance for STS. The success of these models is attributed to their adop- tion of self-supervised learning on text representa- tions, which overcomes the absence of large labeled STS data in many domains. The text representa- tions learned by these models, usually consist of continuous latent variables, and can be fed pairwise ∗Corresponding Author into some functions (e.g. a multi-layer perceptron) to compute their semantic similarity. Compared to the continuous ones, discrete latent variables have drawn much less attention in lan- guage modeling, despite that natural languages are discrete in nature, and there is growing evidence, from several NLP tasks (Jin et al., 2020; Bao et al., 2020), that they are equally good, if not more suit- able, as the continuous counterparts. As a widely- used training technique for learning discrete latent variables, vector-quantized variational autoencoder (VQ-VAE) (Oord et al., 2017) computes the values for the variables through the nearest neighbor look- up across the quantized vectors from the shared latent embedding space. Despite its success in speech recognition and computer vision, VQ-VAE has yet to be investigated in its use and viability in general NLP tasks. In this paper, we intend to ex- plore the use of VQ-VAE and its generated discrete latent variables for …

Content truncated to 500 words