Nightmare at test time: How punctuation prevents parsers from generalizing

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 25–29 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 25 Nightmare at test time: How punctuation prevents parsers from generalizing Anders Søgaard1 Miryam de Lhoneux2 Isabelle Augenstein1 1Dpt. of Computer Science 2 Dpt. of Linguistics and Philology University of Copenhagen Uppsala University Abstract Punctuation is a strong indicator of syntac- tic structure, and parsers trained on text with punctuation often rely heavily on this signal. Punctuation is a diversion, however, since hu- man language processing does not rely on punctuation to the same extent, and in infor- mal texts, we therefore often leave out punc- tuation. We also use punctuation ungrammati- cally for emphatic or creative purposes, or sim- ply by mistake. We show that (a) dependency parsers are sensitive to both absence of punctu- ation and to alternative uses; (b) neural parsers tend to be more sensitive than vintage parsers; (c) training neural parsers without punctuation outperforms all out-of-the-box parsers across all scenarios where punctuation departs from standard punctuation. Our main experiments are on synthetically corrupted data to study the effect of punctuation in isolation and avoid po- tential confounds, but we also show effects on out-of-domain data. 1 Introduction We study the sensitivity of modern dependency parsers to punctuation. While punctuation was originally motivated by reading aloud, serving the purpose of “breath marks” (Baldwin and Coady, 1978), many modern-day punctuation systems are designed to facilitate grammatical disambiguation. This paper aims to show that for this reason, punctuation can significantly hurt the generaliza- tion ability of state-of-the-art syntactic parsers. In other words, syntactic parsers become too reliant on punctuation and therefore suffer from the ab- sence or creative uses of punctuation. Such uses are abundant; see Table 1 for examples from Twit- ter. Such situations, where highly predictive fea- tures are absent or distorted at test time, were re- ferred to in Globerson and Roweis (2006) as night- mare at test time. Human reading is very robust to variation in punctuation (Baldwin and Coady, No punctuation (1) i have so many questions i dont know where to start Creative punctuation (2) What. The. Fuck. Ever. Dot. Com (3) . . . and then , , , , i start to feel ∼lonely∼ Both (4) I feel like ... idk ... idk ... idk man. Nvm I’m good. Table 1: Examples of uses of punctuation 1978); so creative use of punctuation does not hurt human reading performance. In effect, sensitiv- ity to punctuation is a major obstacle that prevents our syntactic parser from achieving human-level robustness. The generalization ability of a dependency parser is usually measured by evaluating its ac- curacy on held-out data, our yardstick to prevent over-fitting, i.e. we define the degree to which a parser has over-fitted to the training data as the difference between performance on training data and performance on the held-out data. This prac- tice is poor when data is not i.i.d., since the held- out data …

Content truncated to 500 words