Global Autoregressive Models for Data-Efficient Sequence Learning

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 900–909 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 900 Global Autoregressive Models for Data-Efficient Sequence Learning Tetiana Parshakova Stanford University⇤ tetianap@stanford.edu Jean-Marc Andreoli Naver Labs Europe {jean-marc.andreoli,marc.dymetman}@naverlabs.com Marc Dymetman Naver Labs Europe Abstract Standard autoregressive seq2seq models are easily trained by max-likelihood, but tend to show poor results under small-data condi- tions. We introduce a class of seq2seq mod- els, GAMs (Global Autoregressive Models), which combine an autoregressive component with a log-linear component, allowing the use of global a priori features to compensate for lack of data. We train these models in two steps. In the first step, we obtain an unnormal- ized GAM that maximizes the likelihood of the data, but is improper for fast inference or eval- uation. In the second step, we use this GAM to train (by distillation) a second autoregres- sive model that approximates the normalized distribution associated with the GAM, and can be used for fast inference and evaluation. Our experiments focus on language modelling un- der synthetic conditions and show a strong per- plexity reduction of using the second autore- gressive model over the standard one. 1 Introduction Neural sequential text generation models have become the standard in NLP applications such as language modelling, NLG, machine transla- tion. When enough data is available, these mod- els can be trained end-to-end with impressive re- sults. Generally, inference and training proceed in an auto-regressive manner, namely, the next decoded symbol is predicted by a locally nor- malized conditional distribution (the “softmax”). This has several advantages: (i) the probability of the sequence is already normalized, by the chain- rule over local decisions, (ii) max-likelihood (ML) training is easy, because the log-likelihood of the full sequence is simply the sum of local CE (cross- entropy) losses, (iii) exact sampling of full se- ⇤ Work conducted during an internship at NAVER Labs Europe. quences from the model distribution is directly ob- tained through a sequence of local sampling deci- sions. However, these autoregressive models (AMs) tend to suffer from a form of myopia. They have difficulty accounting for global properties of the predicted sequences, from overlooking certain as- pects of the semantic input in NLG to duplicating linguistic material or producing “hallucinations” in MT, and generally through being unable to ac- count for long-distance consistency requirements that would be obvious for a human reader.1 The main contributions of this paper are as fol- lows. First, we propose a hybrid seq2seq formaliza- tion, the Global Autoregressive Model (GAM), that combines a local autoregressive component with a global log-linear component, allowing the use of a priori features to compensate for the lack of training data. GAMs are related both to the class of Energy-Based Models (EBM) and to that of Exponential Families (EF), and inherit some important properties from those: an intimate re- lationship between training and sampling (EBM); the identity of empirical and model expectations at maximum-likelihood; convexity of log-likelihood (EF). Second, we propose …

Content truncated to 500 words