An Experiment in Hybrid Dictionary and Statistical Sentence Alignment

An Experiment in Hybrid Dictionary and Statistical Sentence Alignment Nigel Collier, Kenji Ono and Hideki Hirakawa Communication and Information Systems Laboratories Research and Development Center, Toshiba Corporation 1 Komukai Toshiba-cho, Kawasaki-shi, Kanagawa 210-8582, Japan {nigel, ono, hirakawa}©eel, rdc. toshiba, co. j p Abstract The task of aligning sentences in parallel corpora of two languages has been well studied using pure sta- tistical or linguistic models. We developed a linguis- tic method based on lexical snatching with a bilin- gual dictionary and two statistical methods based on sentence length ratios and sentence offset prob- abilities. This paper seeks to further our knowl- edge of the alignment task by comparing the per- formance of the alignment models when used sepa- rately and together, i.e. as a hybrid system. Our results show that for our English-Japanese corpus of newspaper articles, the hybrid system using lexical matching and sentence length ratios outperforms the pure methods. 1 Introduction There have been many approaches proposed to solve the problem of aligning corresponding sentences in parallel corpora. With a few notable exceptions however, much of this work has focussed on ei- ther corpora containing European language pairs or clean-parallel corpora where there is little reformat- ting. In our work we have focussed on developing a method for robust matching of English-Japanese sentences, based primarily on lexical snatching. The method combines statistical information from byte length ratios. We show in this paper that this hybrid model is more effective than its constituent parts used separately. The task of sentence alignment is a critical first step in many automatic applications involving the analysis of bilingual texts such as extraction of bilin- gual vocabulary, extraction of translation templates, word sense disambiguation, word and phrase align- ment, and extraction of parameters for statistical translation models. Many software products which aid human translators now contain sentence align- ment tools as an aid to speeding up editing and ter- minology searching. Various methods have been developed for sentence alignment which we can categorise as either lexical such as (Chen, 1993), based on a large-scale bilin- gual lexicon; statistical such as (Brown et al., 1991) (Church, 1993)(Gale and Church, 1993)(Kay and Rhshcheisen, 1993), based on distributional regular- ities of words or byte-lengdl ratios and possibly in- ducing a bilingual lexicon as a by-product, or hybrid such as (Vtsuro et al., 1994) (Wu, 1994), based on some combination of the other two. Neither of the pure approaches is entirely satisfactory for the fol- lowing reasons: • Text volume limits the usefulness of statistical approaches. We would often like to be able to align small amounts of text, or texts from var- ious domains which do not share the same sta- tistical properties. • Bilingual dictionary coverage limitations mean that we will often encounter problems establish- ing a correspondence in non-general domains. • Dictionary-based approaches are founded on an assumption of lexictd correspondence between language pairs. We cannot always rely on this for non-cognate language pairs, such as English and Japanese. • Texts are often heavily …

Content truncated to 500 words