An Algorithm for Finding Noun Phrase Correspondences in Bilingual Corpora

AN ALGORITHM FOR FINDING NOUN PHRASE CORRESPONDENCES IN BILINGUAL CORPORA Julian Kupiec Xerox Palo Alto Research Center 3333 Coyote Hill Road, Palo Alto, CA kupiec@parc.xerox.com 94304 Abstract The paper describes an algorithm that employs English and French text taggers to associate noun phrases in an aligned bilingual corpus. The tag- gets provide part-of-speech categories which are used by finite-state recognizers to extract simple noun phrases for both languages. Noun phrases are then mapped to each other using an iterative re-estimation algorithm that bears similarities to the Baum-Welch algorithm which is used for train- ing the taggers. The algorithm provides an alter- native to other approaches for finding word cor- respondences, with the advantage that linguistic structure is incorporated. Improvements to the basic algorithm are described, which enable con- text to be accounted for when constructing the noun phrase mappings. INTRODUCTION Areas of investigation using bilingual corpora have included the following: • Automatic sentence alignment [Kay and RSscheisen, 1988, Brown eL al., 1991a, Gale and Church, 1991b]. • Word-sense disambiguation [Dagan el al., 1991, Brown et ai., 1991b, Church and Gale, 1991]. • Extracting word correspondences [Gale and Church, 1991a]. • Finding bilingual collocations [Smadja, 1992]. • Estimating parameters for statistically-based machine translation [Brown et al., 1992]. The work described here makes use of the aligned Canadian Hansards [Gale and Church, 1991b] to obtain noun phrase correspondences be- tween the English and French text. The term "correspondence" is used here to sig- nify a mapping between words in two aligned sen- tences. Consider an English sentence Ei and a French sentence Fi which are assumed to be ap- proximate translations of each other. The sub- script i denotes the i'th alignment of sentences in both languages. A word sequence in E/is defined here as the correspondence of another sequence in Fi if the words of one sequence are considered to represent the words in the other. Single word correspondences have been investi- gated [Gale and Church, 1991a] using a statistic operating on contingency tables. An algorithm for producing collocational correspondences has also been described [Smadja, 1992]. The algorithm in- volves several steps. English collocations are first extracted from the English side of the corpus. In- stances of the English collocation are found and the mutual information is calculated between the instances and various single word candidates in aligned French sentences. The highest ranking candidates are then extended by another word and the procedure is repeated until a corresponding French collocation having the highest mutual in- formation is found. An alternative approach is described here, which employs simple iterative re-estimation. It is used to make correspondences between simple noun phrases that have been isolated in corre- sponding sentences of each language using finite- state recognizers. The algorithm is applicable for finding single or multiple word correspondences and can accommodate additional kinds of phrases. In contrast to the other methods that have been mentioned, the algorithm can be extended in a straightforward way to enable correct correspon- dences to be made in circumstances …

Content truncated to 500 words