Contextualized and Generalized Sentence Representations by Contrastive Self-Supervised Learning: A Case Study on Discourse Relation Analysis

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5578–5584 June 6–11, 2021. ©2021 Association for Computational Linguistics 5578 Contextualized and Generalized Sentence Representations by Contrastive Self-Supervised Learning: A Case Study on Discourse Relation Analysis Hirokazu Kiyomaru and Sadao Kurohashi Graduate School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku, Kyoto, 606-8501, Japan {kiyomaru, kuro}@nlp.ist.i.kyoto-u.ac.jp Abstract We propose a method to learn contextualized and generalized sentence representations us- ing contrastive self-supervised learning. In the proposed method, a model is given a text con- sisting of multiple sentences. One sentence is randomly selected as a target sentence. The model is trained to maximize the similarity be- tween the representation of the target sentence with its context and that of the masked target sentence with the same context. Simultane- ously, the model minimize the similarity be- tween the latter representation and the repre- sentation of a random sentence with the same context. We apply our method to discourse relation analysis in English and Japanese and show that it outperforms strong baseline meth- ods based on BERT, XLNet, and RoBERTa. 1 Introduction Understanding the meaning of a sentence is one of the main interests of natural language process- ing. In recent years, distributed representations are considered to be promising to capture the meaning of a sentence flexibly (Conneau et al., 2017; Arora et al., 2017; Kiros et al., 2015). One typical way to obtain distributed sentence representations is to learn a task that is somehow related to sentence meaning. For example, sen- tence representations trained to solve natural lan- guage inference (Bowman et al., 2015; Williams et al., 2018) are known to be helpful for many language understanding tasks such as sentiment analysis and semantic textual similarity (Conneau et al., 2017; Wieting and Gimpel, 2018; Cer et al., 2018; Reimers and Gurevych, 2019). However, there is an arbitrariness in the choice of tasks used for training. Furthermore, there is a size limitation on manually annotated data, which makes it hard to learn a wide range of language expressions. A solution to these problems is self-supervised learning, which has been used with great suc- cess (Mikolov et al., 2013; Peters et al., 2018; De- vlin et al., 2019). For example, inspired by skip- grams (Mikolov et al., 2013), Kiros et al. (2015) proposed to train a sequence-to-sequence model to generate sentences before and after a sentence, and use the encoder to compute sentence repre- sentations. Inspired by masked language model- ing in BERT, Zhang et al. (2019) and Huang et al. (2020) presented methods to learn contextualized sentence representations through the task of restor- ing a masked sentence from its context. In self-supervised sentence representation learn- ing, sentence generation is typically used as its objective. Such an objective aims to learn a sen- tence representation specific enough to restore the sentence, including minor details. On the other hand, in case we would like to handle the meaning of a larger block such as paragraphs and …

Content truncated to 500 words