Evaluating Grammaticality in Seq2seq Models with a Broad Coverage HPSG Grammar: A Case Study on Machine Translation

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 298–305 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 298 Evaluating Syntactic Properties of Seq2seq Output with a Broad Coverage HPSG: A Case Study on Machine Translation Johnny Tian-Zheng Wei College of Natural Sciences University of Massachusetts Amherst jwei@umass.edu Khiem Pham Department of Computer Science San Jose State University khiem.pham@sjsu.edu Brian Dillon Department of Linguistics University of Massachusetts Amherst brian@linguist.umass.edu Brendan O’Connor College of Information and Computer Sciences University of Massachusetts Amherst brenocon@cs.umass.edu Abstract Sequence to sequence (seq2seq) models are often employed in settings where the target output is natural language. However, the syntactic properties of the language generated from these models are not well understood. We explore whether such output belongs to a formal and realistic grammar, by employing the English Resource Grammar (ERG), a broad coverage, linguistically precise HPSG- based grammar of English. From a French to English parallel corpus, we analyze the parseability and grammatical constructions occurring in output from a seq2seq translation model. Over 93% of the model translations are parseable, suggesting that it learns to generate conforming to a grammar. The model has trouble learning the distribution of rarer syntactic rules, and we pinpoint several constructions that differentiate translations between the references and our model. 1 Introduction Sequence to sequence models (seq2seq; Sutskever et al., 2014; Bahdanau et al., 2014) have found use cases in tasks such as machine translation (Wu et al., 2016), dialogue agents (Vinyals and Le, 2015), and summarization (Rush et al., 2015), where the target output is natural language. How- ever, the decoder side in these models is usually parameterized by gated variants of recurrent neu- ral networks (Hochreiter and Schmidhuber, 1997), and are general models of sequential data not ex- plicitly designed to generate conforming to the grammar of natural language. The syntactic properties of seq2seq output is our central interest. We focus on machine translation as a case study, and situate our work among those French Une situation grotesque. Reference It is a grotesque situation. NMT Output A generic adj situation. root strict sb-hd mc hd-cmp u sp-hd n aj-hdn norm situation grotesque a is hdn bnp-qnt It root frag np frg sp-hd n aj-hdn norm situation generic adj a Figure 1: A test set source-reference pair and the NMT translation. Below are parser derivations in the ERG of both the reference and NMT translation. The ERG is described in §2. Non-syntactic rules have been omitted. The NMT model is trained and tested only on sentence pairs where the reference is parseable by the ERG. The NMT translation may not always be parseable. Analy- sis on model output parseability in §4.1. of artificial language learning, where we train our translation model exclusively on sentence pairs where the target-side output is in our grammar, and test our models by evaluating the output with respect to a grammar. We attempt to understand seq2seq output with the English Resource Gram- mar (Flickinger, 2000), a broad …

Content truncated to 500 words