An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3375–3385 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3375 An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models Hiroshi Noji Artificial Intelligence Research Center AIST, Tokyo, Japan hiroshi.noji@aist.go.jp Hiroya Takamura Artificial Intelligence Research Center AIST, Tokyo, Japan takamura.hiroya@aist.go.jp Abstract We explore the utilities of explicit negative examples in training neural language mod- els. Negative examples here are incorrect words in a sentence, such as barks in *The dogs barks. Neural language models are com- monly trained only on positive examples, a set of sentences in the training data, but re- cent studies suggest that the models trained in this way are not capable of robustly han- dling complex syntactic constructions, such as long-distance agreement. In this paper, we first demonstrate that appropriately using negative examples about particular constructions (e.g., subject-verb agreement) will boost the model’s robustness on them in English, with a negli- gible loss of perplexity. The key to our suc- cess is an additional margin loss between the log-likelihoods of a correct word and an incor- rect word. We then provide a detailed analy- sis of the trained models. One of our findings is the difficulty of object-relative clauses for RNNs. We find that even with our direct learn- ing signals the models still suffer from resolv- ing agreement across an object-relative clause. Augmentation of training sentences involving the constructions somewhat helps, but the ac- curacy still does not reach the level of subject- relative clauses. Although not directly cogni- tively appealing, our method can be a tool to analyze the true architectural limitation of neu- ral models on challenging linguistic construc- tions. 1 Introduction Despite not being exposed to explicit syntactic su- pervision, neural language models (LMs), such as recurrent neural networks, are able to generate fluent and natural sentences, suggesting that they induce syntactic knowledge about the language to some extent. However, it is still under debate whether such induced knowledge about grammar is robust enough to deal with syntactically challeng- ing constructions such as long-distance subject- verb agreement. So far, the results for RNN lan- guage models (RNN-LMs) trained only with raw text are overall negative; prior work has reported low performance on the challenging test cases (Mar- vin and Linzen, 2018) even with the massive size of the data and model (van Schijndel et al., 2019), or argue the necessity of an architectural change to track the syntactic structure explicitly (Wilcox et al., 2019b; Kuncoro et al., 2018). Here the task is to evaluate whether a model assigns a higher likelihood on a grammatically correct sentence (1a) over an incorrect sentence (1b) that is minimally different from the original one (Linzen et al., 2016). (1) a. The author that the guards like laughs. b. * The author that the guards like laugh. In this paper, to obtain a new insight into the syn- tactic abilities of neural LMs, in …

Content truncated to 500 words