WAE_RN: Integrating Wasserstein Autoencoder and Relational Network for Text Sequence

WAE−RN: Integrating Wasserstein Autoencoder and Relational Network for Text Sequence Xinxin Zhang Zhongyuan University of Technology Zhengzhou, China 2018007088 @zut.edu.cn Xiaoming Liu Zhongyuan University of Technology Henan Key Laboratory on Public Opinion Intelligent Analysis ,Zhengzhou, China ming616@zut.edu.cn Guan Yang∗ Zhongyuan University of Technology Zhengzhou, China yangguan @zut.edu.cn Fangfang Li oOh! Media North Sydney, Australia Fangfang.Li @oohmedia.com.au Abstract One challenge in Natural Language Processing (NLP) area is to learn semantic representation in different contexts. Recent works on pre-trained language model have received great atten- tions and have been proven as an effective technique. In spite of the success of pre-trained language model in many NLP tasks, the learned text representation only contains the correlation among the words in the sentence itself and ignores the implicit relationship between arbitrary tokens in the sequence. To address this problem, we focus on how to make our model effectively learn word representations that contain the relational information between any tokens of text se- quences. In this paper, we propose to integrate the relational network(RN) into a Wasserstein autoencoder(WAE). Specifically, WAE and RN are used to better keep the semantic structurse and capture the relational information, respectively. Extensive experiments demonstrate that our proposed model achieves significant improvements over the traditional Seq2Seq baselines. 1 Introduction Sequence problems are common in daily life that involves DNA sequencing in bioinformatics, time series prediction in Information science, and so on. NLP tasks, such as word segmentation, named entity recognition(NER), machine translation(MT), etc, are actually text sequence problems. For text sequence tasks, it is required to predict or generate target sequences based on the understanding of input source sequence, so it plays a pivotal role in NLP to deeply understand the generic knowledge representation in different context. To learn the features of input sequences, probabilistic graphical models, such as Hidden Markov Mod- els(HMM) and Conditional Random Field (CRF), can use manually defined feature functions to trans- form raw data into features, but the quality of the feature functions directly determines the quality of the data presentation. Because deep learning can automatically learn the useful and highly abstract features of the data via artificial neural network(ANN), many researchers devoted themselves to using Neural Networks(NNs) to obtain low dimensional distributed representations of input data, especially in language modeling, using AutoEncoder(AE) (Rumelhart et al., 1988) to retain the text sequence semantic information in different context has shown promising results. These language models are pretrained on large-scale corpus and complex models to obtain the data representation which contains global information and has strong generalization ability, then the latent representation can be adapted to several contexts by fine-tuning them on various tasks. However, these models simply make use of word order information or position information and ignore the implicit relationship between arbitrary tokens in the sequence, resulting in learning inadequately hidden feature representations and obtaining only superficial semantic representation. More recently, studies on attention (Bahdanau et al., 2015; Luong et al., 2015) and self- attention(Klein and Nabi, 2019; Tan et al., 2018) mechanism demonstrate that it …

Content truncated to 500 words