SpanAlign: Sentence Alignment Method based on Cross-Language Span Prediction and ILP

Proceedings of the 28th International Conference on Computational Linguistics, pages 4750–4761 Barcelona, Spain (Online), December 8-13, 2020 4750 SpanAlign: Sentence Alignment Method based on Cross-Language Span Prediction and ILP Katsuki Chousa and Masaaki Nagata and Masaaki Nishino NTT Communication Science Laboratories, NTT Corporation 2-4, Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237, Japan {katsuki.chousa.bg, masaaki.nagata.et, masaaki.nishino.uh}@hco.ntt.co.jp Abstract We propose a novel method of automatic sentence alignment from noisy parallel docu- ments. We first formalize the sentence alignment problem as the independent predictions of spans in the target document from sentences in the source document. We then in- troduce a total optimization method using integer linear programming to prevent span overlapping and obtain non-monotonic alignments. We implement cross-language span prediction by fine-tuning pre-trained multilingual language models based on BERT archi- tecture and train them using pseudo-labeled data obtained from unsupervised sentence alignment method. While the baseline methods use sentence embeddings and assume monotonic alignment, our method can capture the token-to-token interaction between the tokens of source and target text and handle non-monotonic alignments. In sentence alignment experiments on English-Japanese, our method achieved 70.3 F1 scores, which are +8.0 points higher than the baseline method. In particular, our method improved by +53.9 F1 scores for extracting non-parallel sentences. Our method improved the down- stream machine translation accuracy by 4.1 BLEU scores when the extracted bilingual sentences are used for fine-tuning a pre-trained Japanese-to-English translation model. 1 1 Introduction Sentence alignment is the task that automatically extracts parallel sentences from noisy parallel documents. Parallel sentences are used to train cross-language models, especially for machine translation (MT) systems. Both the quantity and quality of the parallel sentences used for train- ing are crucial for developing an accurate neural machine translation (NMT) system (Khayrallah and Koehn, 2018). Recently, automatic sentence alignment methods using neural networks have gained popular- ity (Gr´egoire and Langlais, 2018; Artetxe and Schwenk, 2019a; Yang et al., 2019; Thompson and Koehn, 2019). Such systems have a scoring function to calculate how the two sentences are parallel from sentence embeddings and obtain an alignment hypothesis from these scores with an alignment algorithm. These embeddings are obtained by separately encoding each source and target sentence. However, these prior works do not utilize context information and the interaction between the tokens of the source and target text although it may be useful for calcu- lating the score. Moreover, alignment algorithms of these works assumes monotonic alignments although not every alignment of noisy bilingual documents is monotonic. Quan et al. (2013) described a legislation corpus as an example of non-monotonic alignments. In such cases, the existing methods that assume monotonicity impair the accuracy. In this paper, we propose a novel sentence alignment method, called SpanAlign. That can be decomposed into two methods: scoring and optimization. We formalize the scoring method as a cross-language span prediction model, similar to models for the SQuAD 2.0 question answering task (Rajpurkar et al., 2016). Figure 1 shows an example. 1Our implementation and datasets will be available at https://github.com/nttcslab-nlp/spanalign. This work is licensed under …

Content truncated to 500 words