Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 359–361 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 359 Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis Kelly W. Zhang1 kz918@nyu.edu Samuel R. Bowman1,2,3 bowman@nyu.edu 1Dept. of Computer Science New York University 60 Fifth Avenue New York, NY 10011 2Center for Data Science New York University 60 Fifth Avenue New York, NY 10011 3Dept. of Linguistics New York University 10 Washington Place New York, NY 10003 1 Introduction Recently, researchers have found that deep LSTMs (Hochreiter and Schmidhuber, 1997) trained on tasks like machine translation learn sub- stantial syntactic and semantic information about their input sentences, including part-of-speech (Belinkov et al., 2017a,b; Blevins et al., 2018). These findings begin to shed light on why pre- trained representations, like ELMo and CoVe, are so beneficial for neural language understanding models (Peters et al., 2018; McCann et al., 2017). We still, though, do not yet have a clear under- standing of how the choice of pretraining objec- tive affects the type of linguistic information that models learn. With this in mind, we compare four objectives—language modeling, translation, skip-thought, and autoencoding—on their ability to induce syntactic and part-of-speech informa- tion, holding constant the quantity and genre of the training data, as well as the LSTM architecture. 2 Methodology We control for the data domain by exclusively training on datasets from WMT 2016 (Bojar et al., 2016). We train models on all tasks using the par- allel En-De corpus, which allows us to make fair comparisons across all tasks. We also augment the parallel data with a large monolingual corpus from WMT to examine how the performance of the un- supervised tasks scales with more data. We analyze representations learned by lan- guage models and by the encoders of sequence- to-sequence models.1 Following Belinkov et al. (2017a), after pretraining, we fix the LSTM model parameters and use the hidden states to train aux- iliary classifiers on several probing tasks. We 1All our encoders are 2-layer, bidirectional LSTMs (500- D in each direction)—except for our large forward language models, which are 1000-D and unidirectional. use two syntactic evaluation tasks: part-of-speech (POS) tagging on Penn Treebank WSJ (Marcus et al., 1993) and Combinatorial Categorical Gram- mar (CCG) supertagging on CCG Bank (Hocken- maier and Steedman, 2007). CCG supertagging allows us to measure the degree to which models learn syntactic structure above the word. We also measure how much LSTMs simply memorize in- put sequences with a word identity prediction task. 3 Results Comparing Pretraining Tasks For all pretrain- ing dataset sizes, bidirectional language model (BiLM) and translation encoder representations outperform skip-thought and autoencoder repre- sentations on both POS and CCG tagging. Trans- lation encoders, however, slightly underperform BiLMs, even when both models are trained on the same amount of data. Furthermore, BiLMs trained on the smallest amount of data (1 mil- lion sentences) outperform models trained on all other tasks using …

Content truncated to 500 words