Aligning a Parallel English-Chinese Corpus Statistically With Lexical Criteria

ALIGNING A PARALLEL ENGLISH-CHINESE CORPUS STATISTICALLY WITH LEXICAL CRITERIA Dekai Wu HKUST Department of Computer Science University of Science &: Technology Clear Water Bay, Hong Kong Internet: dekai¢cs.ust.hk Abstract We describe our experience with automatic align- ment of sentences in parallel English-Chinese texts. Our report concerns three related topics: (1) progress on the HKUST English-Chinese Par- allel Bilingual Corpus; (2) experiments addressing the applicability of Gale ~ Church's (1991) length- based statistical method to the task of align- ment involving a non-Indo-European language; and (3) an improved statistical method that also incorporates domain-specific lexical cues. INTRODUCTION Recently, a number of automatic techniques for aligning sentences in parallel bilingual corpora have been proposed (Kay & RSscheisen 1988; Catizone e~ al. 1989; Gale & Church 1991; Brown et al. 1991; Chen 1993), and coarser approaches when sentences are difficult to identify have also been advanced (Church 1993; Dagan e~ al. 1993). Such corpora contain the same material that has been translated by human experts into two lan- guages. The goal of alignment is to identify match- ing sentences between the languages. Alignment is the first stage in extracting structural information and statistical parameters from bilingual corpora. The problem is made more difficult because a sen- tence in one language may correspond to multiple sentences in the other; worse yet, •sometimes sev- eral sentences' content is distributed across multi- ple translated sentences. Approaches to alignment fall into two main classes: lexical and statistical. Le×ically-based techniques use extensive online bilingual lexicons to match sentences. In contrast, statistical tech- niques require almost no prior knowledge and are based solely on the lengths of sentences. The empirical results to date suggest that statistical methods yield performance superior to that of cur- rently available lexical techniques. However, as far as we know, the literature on automatic alignment has been restricted to al- phabetic Indo-European languages. This method- ological flaw weakens the arguments in favor of either approach, since it is unclear to what extent a technique's superiority depends on the similar- ity between related languages. The work reported herein moves towards addressing this problem. 1 In this paper, we describe our experience with automatic alignment of sentences in paral- lel English-Chinese texts, which was performed as part of the SILC machine translation project. Our report concerns three related topics. In the first of the following sections, we describe the objectives of the HKUST English-Chinese Parallel Bilingual Corpus, and our progress. The subsequent sec- tions report experiments addressing the applica- bility of a suitably modified version of Gale & Church's (1991) length-based statistical method to the task of aligning English with Chinese. In the final section, we describe an improved statistical method that also permits domain-specific lexical cues to be incorporated probabilistically. THE ENGLISH-CHINESE CORPUS The dearth of work on non-Indo-European lan- guages can partly be attributed to a lack of the prequisite bilingual corpora. As a step toward remedying this, we are in the process of construct- ing a suitable English-Chinese corpus. To be in- cluded, materials …

Content truncated to 500 words