Learning Topic Representation for SMT with Neural Networks

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 133–143, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Learning Topic Representation for SMT with Neural Networks∗ Lei Cui1, Dongdong Zhang2, Shujie Liu2, Qiming Chen3, Mu Li2, Ming Zhou2, and Muyun Yang1 1School of Computer Science and Technology, Harbin Institute of Technology, Harbin, P.R. China leicui@hit.edu.cn, ymy@mtlab.hit.edu.cn 2Microsoft Research, Beijing, P.R. China {dozhang,shujliu,muli,mingzhou}@microsoft.com 3Shanghai Jiao Tong University, Shanghai, P.R. China simoncqm@gmail.com Abstract Statistical Machine Translation (SMT) usually utilizes contextual information to disambiguate translation candidates. However, it is often limited to contexts within sentence boundaries, hence broader topical information cannot be leveraged. In this paper, we propose a novel approach to learning topic representation for paral- lel data using a neural network architec- ture, where abundant topical contexts are embedded via topic relevant monolingual data. By associating each translation rule with the topic representation, topic rele- vant rules are selected according to the dis- tributional similarity with the source text during SMT decoding. Experimental re- sults show that our method significantly improves translation accuracy in the NIST Chinese-to-English translation task com- pared to a state-of-the-art baseline. 1 Introduction Making translation decisions is a difficult task in many Statistical Machine Translation (SMT) sys- tems. Current translation modeling approaches usually use context dependent information to dis- ambiguate translation candidates. For exam- ple, translation sense disambiguation approaches (Carpuat and Wu, 2005; Carpuat and Wu, 2007) are proposed for phrase-based SMT sys- tems. Meanwhile, for hierarchical phrase-based or syntax-based SMT systems, there is also much work involving rich contexts to guide rule selec- tion (He et al., 2008; Liu et al., 2008; Marton and Resnik, 2008; Xiong et al., 2009). Although these methods are effective and proven successful in many SMT systems, they only leverage within- ∗This work was done while the first and fourth authors were visiting Microsoft Research. sentence contexts which are insufficient in explor- ing broader information. For example, the word driver often means “the operator of a motor ve- hicle” in common texts. But in the sentence “Fi- nally, we write the user response to the buffer, i.e., pass it to our driver”, we understand that driver means “computer program”. In this case, people understand the meaning because of the IT topical context which goes beyond sentence-level analy- sis and requires more relevant knowledge. There- fore, it is important to leverage topic information to learn smarter translation models and achieve better translation performance. Topic modeling is a useful mechanism for dis- covering and characterizing various semantic con- cepts embedded in a collection of documents. At- tempts on topic-based translation modeling in- clude topic-specific lexicon translation models (Zhao and Xing, 2006; Zhao and Xing, 2007), topic similarity models for synchronous rules (Xiao et al., 2012), and document-level translation with topic coherence (Xiong and Zhang, 2013). In addition, topic-based approaches have been used in domain adaptation for SMT (Tam et al., 2007; Su et al., 2012), where they view different topics as different domains. One typical property …

Content truncated to 500 words