Model Invertibility Regularization: Sequence Alignment With or Without Parallel Data

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 609–618, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Model Invertibility Regularization: Sequence Alignment With or Without Parallel Data Tomer Levinboim∗ levinboim.1@nd.edu Ashish Vaswani† avaswani@isi.edu David Chiang∗ dchiang@nd.edu ∗Dept. of Computer Science and Engineering University of Notre Dame †Information Sciences Institute University of Southern California Abstract We present Model Invertibility Regularization (MIR), a method that jointly trains two direc- tional sequence alignment models, one in each direction, and takes into account the invertibil- ity of the alignment task. By coupling the two models through their parameters (as opposed to through their inferences, as in Liang et al.’s Alignment by Agreement (ABA), and Ganchev et al.’s Posterior Regularization (PostCAT)), our method seamlessly extends to all IBM- style word alignment models as well as to alignment without parallel data. Our proposed algorithm is mathematically sound and inher- its convergence guarantees from EM. We eval- uate MIR on two tasks: (1) On word align- ment, applying MIR on fertility based mod- els we attain higher F-scores than ABA and PostCAT. (2) On Japanese-to-English back- transliteration without parallel data, applied to the decipherment model of Ravi and Knight, MIR learns sparser models that close the gap in whole-name error rate by 33% relative to a model trained on parallel data, and further, beats a previous approach by Mylonakis et al. 1 Introduction The transfer of information between languages is a common natural language phenomenon that is in- tuitively invertible. For example, in transliteration, a source-language word is mapped to a target lan- guage’s writing system under a sound preserving mapping (for example, “computer” to Japanese Ro- maji, “konpyutaa”). The original word should then be recoverable from its transliterated version. Simi- larly, in translation, the back-translation of the trans- lation of a word is likely to be that same word itself. In NLP, however, commonly-used generative models describing such phenomena are directional, only concerned with the transfer of source-language symbols to target-language symbols or vice versa, but not both directions. Left unchecked, indepen- dently training two such directional models (source- to-target and target-to-source) often yields two mod- els that diverge from this invertibility intuition. In word alignment, this can lead to disagreements between alignments inferred by a model trained in one direction and those inferred by a model trained in the reverse direction. To remedy this disparity (and other shortcomings), it is common to turn to alignment symmetrization techniques such as grow- diag-final-and (Koehn et al., 2003) which heuristi- cally combines alignments from both directions. Liang et al. (2006) suggest a more fundamental approach they call Alignment by Agreement (ABA), which jointly trains two word alignment models by maximizing their data-likelihoods along with a reg- ularizer that rewards agreement between their align- ment posteriors (computed over each parallel sen- tence pair). Although their EM-like optimization procedure is heuristic, it proves effective at jointly training bidirectional models. Ganchev et al. (2008) propose another approach for …

Content truncated to 500 words