Learning Distributed Word Representations For Bidirectional LSTM Recurrent Neural Network

Proceedings of NAACL-HLT 2016, pages 527–533, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Learning Distributed Word Representations For Bidirectional LSTM Recurrent Neural Network∗ Peilu Wang1,2†, Yao Qian3, Hai Zhao1‡, Frank K. Soong2, Lei He2, Ke Wu1 1Shanghai Jiao Tong University, Shanghai, China 2Microsoft Research Asia, Beijing, China 3Educational Testing Service Research, CA, USA peiluwang@163.com, yqian@ets.org, zhaohai@cs.sjtu.edu.cn {frankkps, helei}@microsoft.com,wuke@cs.sjtu.edu.cn Abstract Bidirectional long short-term memory (BLSTM) recurrent neural network (RNN) has been successfully applied in many tagging tasks. BLSTM-RNN relies on the distributed representation of words, which implies that the former can be futhermore improved through learning the latter better. In this work, we propose a novel approach to learn distributed word representations by training BLSTM-RNN on a specially designed task which only relies on unlabeled data. Our experimental results show that the proposed approach learns useful distributed word representations, as the trained representations significantly elevate the performance of BLSTM-RNN on three tagging tasks: part-of- speech tagging, chunking and named entity recognition, surpassing word representations trained by other published methods. 1 Introduction Distributed word representations represent word with a real valued vector, which is also referred to ∗The work was partially supported by the National Natural Science Foundation of China (Grant No. 61170114, and Grant No. 61272248), the National Basic Research Program of China (Grant No. 2013CB329401), the Science and Technology Com- mission of Shanghai Municipality (Grant No. 13511500200), the European Union Seventh Framework Program (Grant No. 247619), the Cai Yuanpei Program (CSC fund 201304490199, 201304490171), and the art and science interdisciplinary funds of Shanghai Jiao Tong University, No. 14X190040031, and the Key Project of National Society Science Foundation of China, No. 15-ZDA041. †*Work performed as an intern in speech group, Microsoft Research Asia ‡Corresponding author as word embedding. Well learned distributed word representations have been shown capable of captur- ing semantic and syntactic regularities (Pennington et al., 2014a; Mikolov et al., 2013c) and enhanc- ing neural network model by being used as features (Collobert and Weston, 2008; Bengio and Heigold, 2014; Wang et al., 2015). Sequence tagging is a basic structure learning task for natural language processing. Many primary pro- cessing tasks over sentence such as word segmen- tation, named entity recognition and part-of-speech tagging can be formalized as a tagging task (Zhao et al., 2006; Huang and Zhao, 2007; Zhao and Kit, 2008b; Zhao and Kit, 2008a; Zhao et al., 2010; Zhao and Kit, 2011). Recently, many state-of-the-art sys- tems of tagging related tasks are implemented with bidirectional long short-term memory (BLSTM) re- current neural network (RNN), for example, slot filling (Mesnil et al., 2013), part-of-speech tagging (Huang et al., 2015), and dependency parsing (Dyer et al., 2015) etc. All of these systems use distributed representation of words to involve word level infor- mation. Better trained word representations would further improve the state-of-the-art performance of these tasks which makes it worthy to research the training methods of word representations. The existing training methods of word represen- tation can generally be divided into two classes: 1) Matrix …

Content truncated to 500 words