Large-scale Dictionary Construction via Pivot-based Statistical Machine Translation with Significance Pruning and Neural Network Features

Large-scale Dictionary Construction via Pivot-based Statistical Machine Translation with Significance Pruning and Neural Network Features Raj Dabre1, Chenhui Chu2, Fabien Cromieres2, Toshiaki Nakazawa2, Sadao Kurohashi1 1Graduate School of Informatics, Kyoto University 2Japan Science and Technology Agency prajdabre@gmail.com, (chu,fabien,nakazawa)@pa.jst.jp, kuro@i.kyoto-u.ac.jp Abstract We present our ongoing work on large-scale Japanese-Chinese bilingual dictionary con- struction via pivot-based statistical machine translation. We utilize statistical significance pruning to control noisy translation pairs that are induced by pivoting. We construct a large dictionary which we manually verify to be of a high quality. We then use this dictionary and a parallel corpus to learn bilingual neural net- work language models to obtain features for reranking the n-best list, which leads to an ab- solute improvement of 5% in accuracy when compared to a setting that does not use signif- icance pruning and reranking. 1 Introduction Pivot-based statistical machine translation (SMT) (Wu and Wang, 2007) has been shown to be a possi- ble way of constructing a dictionary for the language pairs that have scarce parallel data (Tsunakawa et al., 2009; Chu et al., 2015). The assumption of this method is that there is a pair of large-scale parallel data: one between the source language and an in- termediate resource rich language (henceforth called pivot), and one between that pivot and the target lan- guage. We can use the source-pivot and pivot-target parallel data to develop a source-target term1 trans- lation model for dictionary construction. Pivot-based SMT uses the log linear model as conventional phrase-based SMT (Koehn et al., 2007) does. This method can address the data sparseness problem of directly merging the source-pivot and pivot-target terms, because it can use the portion of terms to generate new terms. Small-scale experi- ments in (Tsunakawa et al., 2009) showed very low 1In this paper, we call the entries in the dictionary terms. A term consists of one or multiple tokens. accuracy of pivot-based SMT for dictionary con- struction.2 This paper presents our study to construct a large- scale Japanese-Chinese (Ja-Zh) scientific dictionary, using large-scale Japanese-English (Ja-En) (49.1M sentences and 1.4M terms) and English-Chinese (En-Zh) (8.7M sentences and 4.5M terms) parallel data via pivot-based SMT. We generate a large pivot translation model using the Ja-En and En-Zh paral- lel data. Moreover, a small direct Ja-Zh translation model is generated using small-scale Ja-Zh parallel data. (680k sentences and 561k terms). Both the direct and pivot translation models are used to trans- late the Ja terms in the Ja-En dictionaries to Zh and the Zh terms in the Zh-En dictionaries to Ja to con- struct a large-scale Ja-Zh dictionary (about 3.6M terms). We address the noisy nature of pivoting large phrase tables by statistical significance pruning (Johnson et al., 2007). In addition, we exploit lin- guistic knowledge of common Chinese characters (Chu et al., 2013) shared in Ja-Zh to further improve the translation model. Large-scale experiments on scientific domain data indicate that our proposed method achieves high quality dictionaries which we manually verify to have a high quality. Reranking the n-best …

Content truncated to 500 words