Controllable Meaning Representation to Text Generation: Linearization and Data Augmentation Strategies

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 555–565, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 555 A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT Masaaki Nagata and Katsuki Chousa and Masaaki Nishino NTT Communication Science Laboratories, NTT Corporation 2-4, Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237, Japan {masaaki.nagata.et,katsuki.chousa.bg,masaaki.nishino.uh}@hco.ntt.co.jp Abstract We present a novel supervised word alignment method based on cross-language span predic- tion. We first formalize a word alignment problem as a collection of independent predic- tions from a token in the source sentence to a span in the target sentence. Since this step is equivalent to a SQuAD v2.0 style question answering task, we solve it using the multi- lingual BERT, which is fine-tuned on manu- ally created gold word alignment data. It is nontrivial to obtain accurate alignment from a set of independently predicted spans. We greatly improved the word alignment accuracy by adding to the question the source token’s context and symmetrizing two directional pre- dictions. In experiments using five word align- ment datasets from among Chinese, Japanese, German, Romanian, French, and English, we show that our proposed method signifi- cantly outperformed previous supervised and unsupervised word alignment methods with- out any bitexts for pretraining. For example, we achieved 86.7 F1 score for the Chinese- English data, which is 13.3 points higher than the previous state-of-the-art supervised method.1 1 Introduction Over the last several years, machine transla- tion accuracy has been greatly improved by neu- ral networks (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Vaswani et al., 2017). However, word align- ment tools, which were developed during the age of statistical machine translation (Brown et al., 1993; Koehn et al., 2007) such as GIZA++ (Och and Ney, 2003), MGIZA (Gao and Vogel, 2008) and FastAlign (Dyer et al., 2013), remain 1Our implementation is available at https://github.com/nttcslab-nlp/word align/ widely used because the improvement of word alignment accuracy has become stagnant. This situation is unfortunate because word alignment could be used for many down- stream tasks including projecting linguistic an- notation (Yarowsky et al., 2001), projecting XML markups (Hashimoto et al., 2019), and enforcing terminology constraints (pre-specified translation) (Song et al., 2019). We could also use it for the user interfaces of post-editing to detect such prob- lems as under-translation (Tu et al., 2016). Word alignment has a long research history. Here, we focus on approaches that use neural networks because they are the state-of-the art. Most previous works that use them for word alignment (Yang et al., 2013; Tamura et al., 2014; Legrand et al., 2016) achieved accuracies that are basically comparable to GIZA++. However, the accuracy of recent works (Garg et al., 2019; Stengel-Eskin et al., 2019; Zenkel et al., 2020) based on the Transformer (Vaswani et al., 2017), which is the state-of-the art neural machine trans- lation model, have started to outperform GIZA++. Garg et al. (2019) made the attention of the Transformer more closely resembled the …

Content truncated to 500 words