Exact Hard Monotonic Attention for Character-Level Transduction

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1530–1537 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1530 Exact Hard Monotonic Attention for Character-Level Transduction Shijie Wu@ and Ryan Cotterell@,H @Department of Computer Science, Johns Hopkins University HDepartment of Computer Science and Technology, University of Cambridge shijie.wu@jhu.edu, rdc42@cam.ac.uk Abstract Many common character-level, string-to- string transduction tasks, e.g. grapheme- to-phoneme conversion and morphological inflection, consist almost exclusively of monotonic transduction. Neural sequence-to- sequence models with soft attention, which are non-monotonic, often outperform popular monotonic models. In this work, we ask the following question: Is monotonicity really a helpful inductive bias in these tasks? We develop a hard attention sequence-to-sequence model that enforces strict monotonicity and learns a latent alignment jointly while learning to transduce. With the help of dynamic pro- gramming, we are able to compute the exact marginalization over all monotonic align- ments. Our models achieve state-of-the-art performance on morphological inflection. Fur- thermore, we find strong performance on two other character-level transduction tasks. Code is available at https://github.com/ shijie-wu/neural-transducer. 1 Introduction Many tasks in natural language can be treated as character-level, string-to-string transduction. The current dominant method is the neural sequence- to-sequence model with soft attention (Bahdanau et al., 2015; Luong et al., 2015). This method has achieved state-of-the-art results in a plethora of tasks, for example, grapheme-to-phoneme conver- sion (Yao and Zweig, 2015), named-entity translit- eration (Rosca and Breuel, 2016) and morpholog- ical inflection generation (Cotterell et al., 2016). While soft attention is very similar to a traditional alignment between the source characters and tar- get characters in some regards, it does not explic- itly a distribution over alignments. On the other hand, neural sequence-to-sequence models with hard alignment (Xu et al., 2015; Wu et al., 2018) are analogous to the latent alignment in the clas- sic IBM models for machine translation, which do model the alignment distribution explicitly (Brown et al., 1993). The standard versions of both soft and hard at- tention are non-monotonic. However, if we look at the data in grapheme-to-phoneme conversion, named-entity transliteration, and morphological inflection—examples are shown in Fig. 1—we see that the tasks require almost exclusively monotonic transduction. Yet, counterintuitively, the state of the art in high resource morphological inflection is held by non-monotonic models (Cotterell et al., 2017)! Indeed, in a recent controlled experiment, Wu et al. (2018) found non-monotonic models (with either soft attention or hard alignment) outperform pop- ular monotonic models (Aharoni and Goldberg, 2017) in the three above mentioned tasks. How- ever, the inductive bias of monotonicity, if correct, should help learn a better model or, at least, learn the same model. In this paper, we hypothesize that the underper- formance of monotonic models stems from the lack of joint training of the alignments with the trans- duction. Generalizing the model of Wu et al. (2018) to enforce monotonic alignments, we show that, for all three tasks considered, monotonicity is a good inductive bias and jointly learning a monotonic …

Content truncated to 500 words