An EBMT system based on word alignment

An EBMT System Based on Word Alignment HOU Hongxu, DENG Dan, ZOU Gang, YU Hongkui, LIU Yang, XIONG Deyi and LIU Qun Institute of Computing Technology, Chinese Academy of Sciences, Beijing, PR China hxhou@ict.ac.cn Abstract This system is an experiment of examples based approach. It is based on a corpus containing 220 thousand sentence pairs with word alignment. The system contains four parts: matching and search, fragment matching, fragment assembling, evaluation and post processing. We use word alignment information to find and combine fragments. 1. Introduction This system is our first experiment of example based approach. It is based on corpus with word alignment. The corpus contains 220 thousands of news, literal, dictionaries, dialog sentence pairs. All sentence pairs are POS tagged and word aligned. 2. System Architecture The system has two parts: corpus and program. The corpus includes 220 thousand sentences pairs and a 460 thousand words and phrases dictionary. The program has four parts: matching and search, fragment matching, fragment assembling, evaluation and post processing. 2.1. Matching and searching Search for the most similar sentences from corpus. 2.2. Fragment matching Find out all matching and non-matching fragment of example sentence and corresponding fragment of translation of example sentence. 2.3. Fragment assembling Assemble the fragment into a full sentence. 2.4. Evaluation Evaluation the result of translation and determine keep or discard a non-aligned part. 2.5. Post processing Process spaces, cases and punctuations. 3. Corpus This system uses a corpus having 220 thousand sentences pair. It contains news, literal, dictionaries, dialog, etc. The sentences in corpus are POS tagged and word aligned. 3.1. POS tag The source language of the corpus is Chinese, and target language is English. For Chinese POS tagging, we use the ICTCLS 2.0. It’s developed by ICT. It uses Multi Layer HMM. In former test of Chinese High Technology Development and Research, it got 97.58% of accuracy and could process 31.5Kb characters per second. 3.2. Word alignment Word alignment is the base of our system. The arithmetic of word alignment is based on dictionary. It uses large scale bilingual dictionary, Word-Net and other human-readable dictionary. It is inspired by Ker’s method [4]. This method mainly depends on similarity measured by bilingual dictionary, relative distortion information and Part-of-Speech information to align words. By setting alignment window it acquires many-to-many word alignments. On a test set of 650 translation sentence pairs of Chinese and English, in which Chinese sentence has 24.8 words in average and English 34.5, the word alignment system gets a result of recall 62.9 ％ at the precision of 84.0%. Our algorithm is improved on Ker’s in these aspects: (1) The computation of relative distortion of Ker is improved, and the initial alignment anchors chosen by dictionary-based word similarity is added to improve alignment. (2) Proposed a concept of ‘alignment window’. By setting alignment window in the aligning process, many-to-many word alignments can be found. 3.3. Dictionary The system uses a 460 thousand words and phrases bilingual dictionary. The dictionary is the base of word …

Content truncated to 500 words