A Discriminative Neural Model for Cross-Lingual Word Alignment

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 910–920, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 910 A Discriminative Neural Model for Cross-Lingual Word Alignment Elias Stengel-Eskin Tzu-Ray Su Matt Post Benjamin Van Durme Johns Hopkins University {elias, tsu6}@jhu.edu {post, vandurme}@cs.jhu.edu Abstract We introduce a novel discriminative word alignment model, which we integrate into a Transformer-based machine translation model. In experiments based on a small number of labeled examples (∼1.7K–5K sentences) we evaluate its performance intrinsically on both English-Chinese and English-Arabic align- ment, where we achieve major improvements over unsupervised baselines (11–27 F1). We evaluate the model extrinsically on data pro- jection for Chinese NER, showing that our alignments lead to higher performance when used to project NER tags from English to Chi- nese. Finally, we perform an ablation analysis and an annotation experiment that jointly sup- port the utility and feasibility of future manual alignment elicitation. 1 Introduction Neural-network-based models for Machine Transla- tion (MT) have set new standards for performance, especially when large amounts of parallel text (bi- text) are available. However, explicit word-to-word alignments, which were foundational to pre-neural statistical MT (SMT) (Brown et al., 1993), have largely been lost in neural MT (NMT) models. This is unfortunate: while alignments are not necessary for NMT systems, they have a wealth of appli- cations in downstream tasks, such as transferring input formatting, incorporating lexica, and human- in-the-loop translation. Crucially, they are cen- tral to cross-lingual dataset creation via projection (Yarowsky et al., 2001), where token-level anno- tations in a high-resource language are projected across alignments to a low-resource language; us- ing projection, datasets have been created for a vari- ety of natural language processing tasks, including named-entity recognition (NER), part-of-speech tagging, parsing, information extraction (IE), and semantic role labeling (Yarowsky et al., 2001; Hwa President Zemin Jiang will leave Beijing ! " # $ % & ' ... ( )... * + , -. Figure 1: One application of alignments is to project annotations (e.g., Named Entities) across bitexts. Here we see a lack of correspondence between FastAlign (red) alignments and gold alignments (blue). et al., 2005; Riloff et al., 2002; Pad´o and Lapata, 2009; Xi and Hwa, 2005). This paradigm allows researchers to address a lack of annotated multilin- gual resources and changing ontologies with mini- mal annotation effort. The introduction of attention (Bahdanau et al., 2014) has allowed NMT to advance past decoding from a single sentence-level vector representation and has been crucial in supporting fluent, mean- ingful translations, especially of longer sentences. However, despite their intuitive analogy to word alignments, attention-based alignments typically do not correspond very closely to gold-standard human annotations and suffer from systematic er- rors (Koehn and Knowles, 2017; Ghader and Monz, 2017), falling short of the promise to jointly learn to align and translate. This problem has only been exacerbated by the introduction of deeper models containing multiple attention layers and heads …

Content truncated to 500 words