Reliable Measures for Aligning Japanese-English News Articles and Sentences

Reliable Measures for Aligning Japanese-English News Articles and Sentences Masao Utiyama and Hitoshi Isahara Communications Research Laboratory 3-5 Hikari-dai, Seika-cho, Souraku-gun, Kyoto 619-0289 Japan mutiyama@crl.go.jp and isahara@crl.go.jp Abstract We have aligned Japanese and English news articles and sentences to make a large parallel corpus. We first used a method based on cross-language informa- tion retrieval (CLIR) to align the Japanese and English articles and then used a method based on dynamic programming (DP) matching to align the Japanese and English sentences in these articles. How- ever, the results included many incorrect alignments. To remove these, we pro- pose two measures (scores) that evaluate the validity of alignments. The measure for article alignment uses similarities in sentences aligned by DP matching and that for sentence alignment uses similar- ities in articles aligned by CLIR. They enhance each other to improve the accu- racy of alignment. Using these measures, we have successfully constructed a large- scale article and sentence alignment cor- pus available to the public. 1 Introduction A large-scale Japanese-English parallel corpus is an invaluable resource in the study of natural language processing (NLP) such as machine translation and cross-language information retrieval (CLIR). It is also valuable for language education. However, no such corpus has been available to the public. We recently have obtained a noisy parallel cor- pus of Japanese and English newspapers consisting of issues published over more than a decade and have tried to align their articles and sentences. We first aligned the articles using a method based on CLIR (Collier et al., 1998; Matsumoto and Tanaka, 2002) and then aligned the sentences in these articles by using a method based on dynamic programming (DP) matching (Gale and Church, 1993; Utsuro et al., 1994). However, the results included many in- correct alignments due to noise in the corpus. To remove these, we propose two measures (scores) that evaluate the validity of article and sen- tence alignments. Using these, we can selectively extract valid alignments. In this paper, we first discuss the basic statistics on the Japanese and English newspapers. We next explain methods and measures used for alignment. We then evaluate the effectiveness of the proposed measures. Finally, we show that our aligned corpus has attracted people both inside and outside the NLP community. 2 Newspapers Aligned The Japanese and English newspapers used as source data were the Yomiuri Shimbun and the Daily Yomiuri. They cover the period from September 1989 to December 2001. The number of Japanese articles per year ranges from 100,000 to 350,000, while English articles ranges from 4,000 to 13,000. The total number of Japanese articles is about 2,000,000 and the total number of English articles is about 110,000. The number of English articles rep- resents less than 6 percent that of Japanese articles. Therefore, we decided to search for the Japanese ar- ticles corresponding to each of the English articles. The English articles as of mid-July 1996 have tags indicating whether they are translated from Japanese articles or not, though they don’t have …

Content truncated to 500 words