Word Ordering as Unsupervised Learning Towards Syntactically Plausible Word Representations

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 70–79, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Word Ordering as Unsupervised Learning Towards Syntactically Plausible Word Representations Noriki Nishida and Hideki Nakayama Graduate School of Information Science and Technology The University of Tokyo {nishida,nakayama}@nlab.ci.i.u-tokyo.ac.jp Abstract The research question we explore in this study is how to obtain syntacti- cally plausible word representations with- out using human annotations. Our un- derlying hypothesis is that word order- ing tests, or linearizations, is suitable for learning syntactic knowledge about words. To verify this hypothesis, we de- velop a differentiable model called Word Ordering Network (WON) that explic- itly learns to recover correct word order while implicitly acquiring word embed- dings representing syntactic knowledge. We evaluate the word embeddings pro- duced by the proposed method on down- stream syntax-related tasks such as part- of-speech tagging and dependency pars- ing. The experimental results demonstrate that the WON consistently outperforms both order-insensitive and order-sensitive baselines on these tasks. 1 Introduction Distributed word representations have been suc- cessfully utilized to transfer lexical knowledge to downstream tasks in a semi-supervised man- ner, and well known to benefit various applica- tions (Turian et al., 2010; Collobert et al., 2011; Socher et al., 2011). As different applications gen- erally require different features, it is crucial to choose representations suitable for target down- stream tasks. The research question we want to explore in this study is how to obtain syntactically plausi- ble word representations without human annota- tions, with a focus on syntax-related tasks (pars- ing, etc.). Whereas a variety of approaches re- lated to semantic word embeddings have been pro- Figure 1: Illustration of the word ordering task. The goal of the word ordering task is to recover an original order given a set of shuffled tokens. The figure shows an example where original sentence is “this is a short sentence.” To correctly reorder the tokens, syntactic knowledge about words (e.g. grammatical classes of words and their possible relations) is indispensable. In this study, we ex- plore how well the word ordering task can be an objective to obtain syntactic word representations. posed (Mikolov et al., 2013a,b; Pennington et al., 2014), it still remains unclear how we should ob- tain syntactic word embeddings from unannotated corpora. Word ordering tests, or linearizations, are com- monly used to evaluate students’ language profi- ciency. Suppose that we are given a set of ran- domly shuffled tokens {“a”, “is,” “sentence,” “short,” “this,” “.”}. In this case we can easily recover the original order: “this is a short sen- tence.” We consider this doable thanks to our knowledge about grammatical classes (e.g., part- of-speech (POS) tags) of words and their possible relations. We depict the above explanation in Fig- ure 1. Of course, it might not be necessary for ma- chines to mimic exactly the same reasoning pro- 70 cess in humans. However, syntactic knowledge about words is crucial for both humans and ma- chines to solve the …

Content truncated to 500 words