Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4208–4220 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4208 Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels Harris Chan∗ Vector Institute University of Toronto hchan@cs.toronto.edu Jamie Kiros Google Research, Brain Team kiros@google.com William Chan Google Research, Brain Team williamchan@google.com Abstract A channel corresponds to a viewpoint or trans- formation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning, but through two separate channels corresponding to their languages. In this work, we present the Multichannel Generative Language Model (MGLM). MGLM is a generative joint distri- bution model over channels. MGLM marginal- izes over all possible factorizations within and across all channels. MGLM endows flexible inference, including unconditional generation, conditional generation (where 1 channel is ob- served and other channels are generated), and partially observed generation (where incom- plete observations are spread across all the channels). We experiment with the Multi30K dataset containing English, French, Czech, and German. We demonstrate experiments with unconditional, conditional, and partially conditional generation. We provide qualita- tive samples sampled unconditionally from the generative joint distribution. We also quantita- tively analyze the quality-diversity trade-offs and find MGLM outperforms traditional bilin- gual discriminative models. 1 Introduction A natural way to consider two parallel sentences in different languages is that each language expresses the same underlying meaning from a different view- point. Each language can be thought of as a trans- formation that maps an underlying concept into a view that we collectively agree is determined as ‘English’ or ‘French’. Similarly, an image of a cat and the word ‘cat’ are expressing two views of the same underlying concept. In this case, the image corresponds to a high bandwidth channel and the word ‘cat’ to a low bandwidth channel. This way of conceptualizing parallel viewpoints naturally ∗Work done during an internship at Google Brain. leads to the formulation of a fully generative model over each instance, where the transformation corre- sponds to a particular generation of the underlying view. We define each of these views as a channel. As a concrete example, given a parallel corpus of English and French sentences, English and French become two channels, and the corresponding gen- erative model becomes p(English, French). One key advantage of this formulation is that a single model can be trained to capture the full expressivity of the underlying concept, allowing us to compute conditionals and marginals along with the joint. In parallel sentences, the conditionals correspond to translations from one channel to another while the marginals correspond to standard monolingual language models. In this work, we present a general framework for modeling the joint distribution p(x1, ..., xk) over k channels by marginalizing over all possi- ble factorizations across the channels and within each channel. This formulation allows our frame- work to perform: 1) unconditional generation, 2) fully conditional generation (source channels are fully observed and fixed), and 3) partial conditional generation (source channels …

Content truncated to 500 words