Learning to Organize a Bag of Words into Sentences with Neural Networks: An Empirical Study

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1682–1691 June 6–11, 2021. ©2021 Association for Computational Linguistics 1682 Learning to Organize a Bag of Words into Sentences with Neural Networks: An Empirical Study Chongyang Tao1, Shen Gao1, Juntao Li1, Yansong Feng1, Dongyan Zhao1∗, Rui Yan1,2,3∗ 1Wangxuan Institute of Computer Technology, Peking University 2Gaoling School of Artificial Intelligence, Renmin University of China 3Beijing Academy of Artificial Intelligence 1{chongyangtao,shengao,lijuntao,fengyansong,zhaody}@pku.edu.cn 2{ruiyan}@ruc.edu.cn Abstract Sequential information, a.k.a., orders, is as- sumed to be essential for processing a se- quence with recurrent neural network or con- volutional neural network based encoders. However, is it possible to encode natural lan- guages without orders? Given a bag of words from a disordered sentence, humans may still be able to understand what those words mean by reordering or reconstructing them. Inspired by such an intuition, in this paper, we perform a study to investigate how “order” information takes effects in natural language learning. By running comprehensive comparisons, we quan- titatively compare the ability of several repre- sentative neural models to organize sentences from a bag of words under three typical scenar- ios, and summarize some empirical findings and challenges, which can shed light on future research on this line of work. 1 Introduction Though significant progress has been made, it is still mysterious how humans are able to under- stand, organize, and generate natural languages. In the field of natural language processing, many efforts have been made to enhance computa- tional models. Recently, recurrent neural net- works (Mikolov et al., 2010) and encoder-decoder architectures (Sutskever et al., 2014) with long short-term memory (Hochreiter and Schmidhuber, 1997) and gated recurrent unit (Chung et al., 2014) have demonstrated state-of-the-art performance in sequence modeling and generation. Nowadays, the encoder-decoder architectures have become a widely used approach for sequence- to-sequence tasks such as machine translation (Bah- danau et al., 2015), text summarization (Paulus et al., 2018) and dialogue generation (Serban et al., 2016). Such models generally encode the input sequence into a vector representation using recur- rent neural networks (RNNs) (Sutskever et al., ∗Corresponding authors: Dongyan Zhao and Rui Yan. 2014), convolutional neural networks (Gehring et al., 2017) or transformer architectures (Vaswani et al., 2017). The decoder then produces the out- put sequence step-by-step, conditioned on the en- codings of the encoder. Basically, those encoders process information along the sentence sequences, where sequential information is recurrently mod- eled at each position of the sequences. Thus these models are sensitive to word orders. Moreover, it has been demonstrated that order matters in se- quence encoding (Vinyals et al., 2015). Admittedly yes, order information is important for sequences learning and encoding. An interesting question might be that, is it possible to encode natural lan- guages without considering order information? Take a look at an example of word rearrange quizzes for language learners1. Given a bag of words from a disordered sentence {the dog James talking sat next to himself to .}, most people …

Content truncated to 500 words