Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 590–599 August 1–6, 2021. ©2021 Association for Computational Linguistics 590 OoMMix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification Seonghyeon Lee1, Dongha Lee2 and Hwanjo Yu1∗ 1Dept. of Computer Science and Engineering, POSTECH, Republic of Korea 2Institute of Artificial Intelligence, POSTECH, Republic of Korea {sh0416,dongha.lee,hwanjoyu}@postech.ac.kr Abstract Recent studies on neural networks with pre- trained weights (i.e., BERT) have mainly fo- cused on a low-dimensional subspace, where the embedding vectors computed from input words (or their contexts) are located. In this work, we propose a new approach, called OoMMix, to finding and regularizing the re- mainder of the space, referred to as out-of- manifold, which cannot be accessed through the words. Specifically, we synthesize the out- of-manifold embeddings based on two embed- dings obtained from actually-observed words, to utilize them for fine-tuning the network. A discriminator is trained to detect whether an in- put embedding is located inside the manifold or not, and simultaneously, a generator is opti- mized to produce new embeddings that can be easily identified as out-of-manifold by the dis- criminator. These two modules successfully collaborate in a unified and end-to-end manner for regularizing the out-of-manifold. Our ex- tensive evaluation on various text classification benchmarks demonstrates the effectiveness of our approach, as well as its good compatibil- ity with existing data augmentation techniques which aim to enhance the manifold. 1 Introduction Neural networks with a word embedding table have been the most popular approach to a wide range of NLP applications. The great success of transformer-based contextual embeddings as well as masked language models (Devlin et al., 2019; Liu et al., 2019b; Raffel et al., 2020) makes it pos- sible to exploit the pre-trained weights, fully opti- mized by using large-scale corpora, and it brought a major breakthrough to many problems. For this reason, most recent work on text classification has achieved state-of-the-art performances by fine- tuning the network initialized with the pre-trained ∗ Corresponding author weight (Devlin et al., 2019). However, they suf- fer from extreme over-parameterization due to the large pre-trained weight, which allows them to be easily overfitted to its relatively small training data. Along with outstanding performances of the pre- trained weight, researchers have tried to reveal the underlying structure encoded in its embedding space (Rogers et al., 2021). One of the important findings is that the contextual embeddings com- puted from words usually form a low-dimensional manifold (Ethayarajh, 2019). In particular, a quan- titative analysis on the space (Cai et al., 2021), which measured the effective dimension size of BERT after applying PCA on its contextual em- bedding vectors, showed that 33% of dimensions covers 80% of the variance. In other words, only the low-dimensional subspace is utilized for fine- tuning BERT, although a high-dimensional space (i.e., model weights with a high capacity) is pro- vided for training. Based on this finding on contex- tual embedding space, we aim …

Content truncated to 500 words