Efficient Bilingual Generalization from Neural Transduction Grammar Induction

Efficient Bilingual Generalization from Neural Transduction Grammar Induction Yuchen YAN, Dekai WU, Serkan KUMYOL Department of Computer Science and Engineering, Human Language Technology Center, The Hong Kong University of Science and Technology {yyanaa|dekai|skumyol}@cs.ust.hk Abstract We introduce (1) a novel neural network structure for bilingual modeling of sentence pairs that allows effi- cient capturing of bilingual relationship via biconstituent composition, (2) the concept of neural network bipars- ing, which applies to not only machine translation (MT) but also to a variety of other bilingual research areas, and (3) the concept of a biparsing-backpropagation train- ing loop, which we hypothesize that can efficiently learn complex biparse tree patterns. Our work distinguishes from sequential attention-based models, which are more traditionally found in neural machine translation (NMT) in three aspects. First, our model enforces composi- tional constraints. Second, our model has a smaller search space in terms of discovering bilingual relationships from bilingual sentence pairs. Third, our model produces ex- plicit biparse trees, which enable transparent error anal- ysis during evaluation and external tree constraints dur- ing training. 1. Introduction In this paper, we introduce a neural network structure for modeling of bilingual sentence pairs that features efficient capturing of bilingual relationships by learn- ing explicit compositional biconstituent structures, as opposed to conventional attention-based NMT models, which learn flat token-to-token bilingual relationships requiring numerous parallel corpora. Token-to-token bilingual relationship formalism is inefficient in two as- pects. First, it lacks compositional structure, which is the key to generalizing biphrases from bitokens and gen- eralizing bisentences from biphrases. Second, there are no constraints on the space of all possible token align- ments, resulting in the attention layer inefficiently ex- ploring strategys for such a huge search space. Our model skips directly to the compositional structure, represent- ing bilingual relationships by recursively composing bi- constituents with one extra degree of ordering flexibil- ity. We propose a new training strategy based on what we call a biparsing-backpropagation training loop, inspired by our hypothesis that good biparse trees lead to better models, and better models will compute bet- ter biparse trees, forming a faster feedback loop to ef- ficiently capture bilingual relationships in low resource scenarios. When biparsing a corpus, desirable tree pat- terns tend to show up repeatedly because they explain more bilingual phenomenon, which will be learned dur- ing backpropagation. In the next epoch, these learned patterns will help compute more accurate biparse trees, revealing more complex and desirable tree patterns. The paper is divided into two main parts. We begin in the first part below by laying out the basic formal- ism of soft transduction grammars and soft biparse trees, which are the neural network analogous to sym- bolic transduction grammars and symbolic biparse trees. We then introduce several competitive neural network designs and explains how the design decisions we make are suitable in terms of the generalizablity of the neural network structure and the expressiveness of the trans- duction grammar. In the second part, we explain how our neural network model implements the formalism of …

Content truncated to 500 words