Simultaneous Sentence Boundary Detection and Alignment with Pivot-based Machine Translation Generated Lexicons

Simultaneous Sentence Boundary Detection and Alignment with Pivot-based Machine Translation Generated Lexicons Antoine Bourlon1, Chenhui Chu1, Toshiaki Nakazawa1, Sadao Kurohashi2 1Japan Science and Technology Agency 2Graduate School of Informatics, Kyoto University E-mail: {bourlon, chu, nakazawa}@pa.jst.jp, kuro@i.kyoto-u.ac.jp Abstract Sentence alignment is a task that consists in aligning the parallel sentences in a translated article pair. This paper describes a method to perform sentence boundary detection and alignment simultaneously, which significantly improves the alignment accuracy on languages like Chinese with uncertain sentence boundaries. It relies on the definition of hard (certain) and soft (uncertain) punctuation delimiters, the latter being possibly ignored to optimize the alignment result. The alignment method is used in combination with lexicons automatically generated from the input article pairs using pivot-based MT, achieving better coverage of the input words with fewer entries than pre-existing dictionaries. Pivot-based MT makes it possible to build dictionaries for language pairs that have scarce parallel data. The alignment method is implemented in a tool that will be freely available in the near future. Keywords: Sentence Alignment, Sentence Segmentation, Pivot-based Machine Translation 1. Introduction Sentence alignment is a task that consists in aligning the parallel sentences in a translated article pair, which are cru- cial for machine translation (MT) (Koehn et al., 2003). Pre- vious studies first split the source and target articles into sentences respectively using punctuation information, and then align the source and target sentences based on sentence length and/or bilingual lexicons (Ma, 2006). However, the monolingually determined sentence boundaries are not op- timized for sentence alignment, because translation equiv- alents might cross the monolingual sentence boundaries. In this paper, we propose a method to perform sentence boundary detection and alignment simultaneously, which significantly improves the alignment accuracy. Sentence alignment methods are generally based on two kinds of algorithms: length-based algorithms align sen- tences according solely to their lengths (Brown et al., 1991), while lexicon-based algorithms use lexical informa- tion to calculate similarity between source and target sen- tences (Ma, 2006). Length-based algorithms are typically faster but not suited for processing noisy corpus (corpus containing article pairs with omitted or wrong translations). Lexicon-based algorithms, like the one used in Champol- lion (Ma, 2006)1 are more robust. However their perfor- mance highly depends on the coverage and quality of the lexicon, and large-scale lexicon with high quality is not easy to obtain especially for low resource language pairs. In this paper, we propose to use lexicons generated by a pivot- based MT system, which could be constructed even for low resource languages (Dabre et al., 2015). Experiments con- ducted on Chinese-Japanese scientific articles verify the ef- fectiveness of our proposed method. 2. Simultaneous Sentence Boundary Detection and Alignment Our proposed alignment method consists in the following steps 1http://champollion.sourceforge.net 1. Split source and target articles into sentence candi- dates. 2. Normalize words in sentence candidates to maximize matching rate between words in source, target, and lexicons. 3. Compute alignment path with the highest similarity between source and target. An alignment path is com- posed of …

Content truncated to 500 words