Word Alignment Modeling with Context Dependent Deep Neural Network

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 166–175, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Word Alignment Modeling with Context Dependent Deep Neural Network Nan Yang1, Shujie Liu2, Mu Li2, Ming Zhou2, Nenghai Yu1 1University of Science and Technology of China, Hefei, China 2Microsoft Research Asia, Beijing, China {v-nayang,shujliu,muli,mingzhou}@microsoft.com ynh@ustc.edu.cn Abstract In this paper, we explore a novel bilin- gual word alignment approach based on DNN (Deep Neural Network), which has been proven to be very effective in var- ious machine learning tasks (Collobert et al., 2011). We describe in detail how we adapt and extend the CD-DNN- HMM (Dahl et al., 2012) method intro- duced in speech recognition to the HMM- based word alignment model, in which bilingual word embedding is discrimina- tively learnt to capture lexical translation information, and surrounding words are leveraged to model context information in bilingual sentences. While being ca- pable to model the rich bilingual corre- spondence, our method generates a very compact model with much fewer parame- ters. Experiments on a large scale English- Chinese word alignment task show that the proposed method outperforms the HMM and IBM model 4 baselines by 2 points in F-score. 1 Introduction Recent years research communities have seen a strong resurgent interest in modeling with deep (multi-layer) neural networks. This trending topic, usually referred under the name Deep Learning, is started by ground-breaking papers such as (Hin- ton et al., 2006), in which innovative training pro- cedures of deep structures are proposed. Unlike shallow learning methods, such as Support Vector Machine, Conditional Random Fields, and Maxi- mum Entropy, which need hand-craft features as input, DNN can learn suitable features (represen- tations) automatically with raw input data, given a training objective. DNN did not achieve expected success until 2006, when researchers discovered a proper way to intialize and train the deep architectures, which contains two phases: layer-wise unsupervised pre- training and supervised fine tuning. For pre- training, Restricted Boltzmann Machine (RBM) (Hinton et al., 2006), auto-encoding (Bengio et al., 2007) and sparse coding (Lee et al., 2007) are pro- posed and popularly used. The unsupervised pre- training trains the network one layer at a time, and helps to guide the parameters of the layer towards better regions in parameter space (Bengio, 2009). Followed by fine tuning in this region, DNN is shown to be able to achieve state-of-the-art per- formance in various area, or even better (Dahl et al., 2012) (Kavukcuoglu et al., 2010). DNN also achieved breakthrough results on the ImageNet dataset for objective recognition (Krizhevsky et al., 2012). For speech recognition, (Dahl et al., 2012) proposed context-dependent neural network with large vocabulary, which achieved 16.0% rel- ative error reduction. DNN has also been applied in Natural Lan- guage Processing (NLP) field. Most works con- vert atomic lexical entries into a dense, low di- mensional, real-valued representation, called word embedding; Each dimension represents a latent as- pect of a word, capturing its semantic and syntac- tic properties (Bengio …

Content truncated to 500 words