Bilingual Autoencoders with Global Descriptors for Modeling Parallel Sentences

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2548–2558, Osaka, Japan, December 11-17 2016. Bilingual Autoencoders with Global Descriptors for Modeling Parallel Sentences Biao Zhang1, Deyi Xiong2∗, Jinsong Su1, Hong Duan1 and Min Zhang2 Xiamen University, Xiamen, China 3610051 Soochow University, Suzhou, China 2150062 zb@stu.xmu.edu.cn, {jssu,hduan}@xmu.edu.cn {dyxiong, minzhang}@suda.edu.cn Abstract Parallel sentence representations are important for bilingual and cross-lingual tasks in natural lan- guage processing. In this paper, we explore a bilingual autoencoder approach to model parallel sentences. We extract sentence-level global descriptors (e.g. min, max) from word embeddings, and construct two monolingual autoencoders over these descriptors on the source and target lan- guage. In order to tightly connect the two autoencoders with bilingual correspondences, we force them to share the same decoding parameters and minimize a corpus-level semantic distance be- tween the two languages. Being optimized towards a joint objective function of reconstruction and semantic errors, our bilingual antoencoder is able to learn continuous-valued latent represen- tations for parallel sentences. Experiments on both intrinsic and extrinsic evaluations on statisti- cal machine translation tasks show that our autoencoder achieves substantial improvements over the baselines. 1 Introduction Neural sentence modeling that learns continuous-valued vector representations for sentences in a low- dimensional latent semantic space, has recently attracted considerable interests in the field of nature language processing (NLP). A variety of models have been proposed (Collobert and Weston, 2008; Socher et al., 2011; Mikolov et al., 2011; Hermann and Blunsom, 2013; Kalchbrenner and Blunsom, 2013; Kalchbrenner et al., 2014; Kim, 2014; Ma et al., 2015; Tai et al., 2015; Zhang et al., 2015a). Most of these models, however, focus on monolingual cases where sentences from a single language are modeled. They do not explore semantic correspondences among parallel sentences. On account of this, monolingual neural sentence models are not naturally fit for bilingual or cross-lingual NLP tasks, such as machine translation, cross-lingual classification and information retrieval. In order to induce sentence representations in a bilingual rather than monolingual semantic space, re- searchers have proposed a few approaches, following efforts of bilingual word embeddings (Klementiev et al., 2012; Zou et al., 2013). These studies either explore recursive/recurrent neural networks (Zhang et al., 2014; Su et al., 2015) or bag-of-words based neural networks (Yih et al., 2011; Chandar et al., 2014; Lauly et al., 2014; Hermann and Blunsom, 2014; Zhou et al., 2015) to learn bilingual sentence repre- sentations. The former recursively compose sentences from the bottom up, taking into account bilingual constraints from word alignments. Due to the complexity, they are not easy to be scalable. Additionally, they also suffer from errors and noises of word alignments. In contrast, the latter are relatively simple and scalable. However, they often heavily rely on only one descriptor of the bag-of-words embeddings (e.g. the avg representation) and hence are weak in capturing fine-grained complex linguistic phenomena. Therefore we believe that modeling parallel sentences still remains a serious challenge. Inspired by works on multimodal autoencoders (Ngiam et al., 2011; Wang et al., …

Content truncated to 500 words