WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11084–11099 July 9-14, 2023 ©2023 Association for Computational Linguistics WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction Qiyu Wu1, Masaaki Nagata2, Yoshimasa Tsuruoka1 1The University of Tokyo, Tokyo, Japan 2NTT Communication Science Laboratories, NTT Corporation, Kyoto, Japan 1{qiyuw, yoshimasa-tsuruoka}@g.ecc.u-tokyo.ac.jp 2masaaki.nagata@ntt.com Abstract Most existing word alignment methods rely on manual alignment datasets or parallel corpora, which limits their usefulness. Here, to mitigate the dependence on manual data, we broaden the source of supervision by relaxing the re- quirement for correct, fully-aligned, and paral- lel sentences. Specifically, we make noisy, par- tially aligned, and non-parallel paragraphs. We then use such a large-scale weakly-supervised dataset for word alignment pre-training via span prediction. Extensive experiments with various settings empirically demonstrate that our approach, which is named WSPAlign, is an effective and scalable way to pre-train word aligners without manual data. When fine-tuned on standard benchmarks, WSPAlign has set a new state of the art by improving upon the best supervised baseline by 3.3~6.1 points in F1 and 1.5~6.1 points in AER . Furthermore, WSPAlign also achieves competitive perfor- mance compared with the corresponding base- lines in few-shot, zero-shot and cross-lingual tests, which demonstrates that WSPAlign is potentially more practical for low-resource lan- guages than existing methods. 1 1 Introduction Word alignment, which aims to align the cor- responding words in parallel texts, is a funda- mental Natural Language Processing (NLP) task that was originally developed as an important supporting task for statistical machine transla- tion. While deep end-to-end models have be- come the mainstream solution for machine transla- tion, word alignment is still of great importance in many NLP scenarios, such as projecting lin- guistic annotations (David et al., 2001) and XML markups (Hashimoto et al., 2019), post-editing for detecting problem of under-translation (Tu et al., 2016), and enforcing pre-specified terminology 1The source code is publicly available at https://github.com/qiyuw/wspalign. constraints in translation (Song et al., 2019). Be- sides, word alignment can also improve the cross- lingual language pre-training (Chi et al., 2021). However, most existing word alignment methods rely on either manual alignment datasets or parallel corpora for training, which weakens their useful- ness because of the limiting accessibility of data. An additional weakness with requiring manual data is the generalization ability because deep mod- els trained on a dataset can fail on other datasets. Therefore, these existing approaches are also lim- ited in terms of potential cross-lingual use. On the other hand, recent studies (Mahajan et al., 2018; Kolesnikov et al., 2020; Chen et al., 2021; Galvez et al., 2021; Radford et al., 2022) in various fields leverage weak supervision signals in large-scale data available on the web for pre-training, which is a promising alternative to training on manual data. Inspired by this, we propose to utilize Wikipedia and multi-lingual Pre-trained Language Models (PLMs) to make large-scale word alignment super- vision for pre-training via span prediction. We broaden the source of supervision by relaxing …

Content truncated to 500 words