Linguistic representations in multi-task neural networks for ellipsis resolution

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 66–73 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 66 Linguistic representations in multi-task neural networks for ellipsis resolution Ola Rønning Cobiro ronning@pm.me Daniel Hardt Copenhagen Business School dh.msc@cbs.dk Anders Søgaard University of Copenhagen soegaard@di.ku.dk Abstract Sluicing resolution is the task of identifying the antecedent to a question ellipsis. An- tecedents are often sentential constituents, and previous work has therefore relied on syntac- tic parsing, together with complex linguistic features. A recent model instead used partial parsing as an auxiliary task in sequential neu- ral network architectures to inject syntactic in- formation. We explore the linguistic informa- tion being brought to bear by such networks, both by defining subsets of the data exhibit- ing relevant linguistic characteristics, and by examining the internal representations of the network. Both perspectives provide evidence for substantial linguistic knowledge being de- ployed by the neural networks. 1 Introduction Sluices are questions where material beyond the wh-word is missing and must be retrieved from context. Consider the following example from Rønning et al. (2018): (1) If [this is not practical], explain why. Here, the antecedent is the complete sentential constituent, this is not practical. Anand and Hardt (2016) present a sluice res- olution system, in which candidate antecedents are required to be sentential constituents. Fur- thermore, each candidate is represented by fea- tures manually defined over syntactic dependency structures. Anand and Hardt report an accuracy of antecedent selection of 0.72, and a token-level F1 score 0.72, applied to a dataset based on news content (Anand and McCloskey, 2015). Rønning et al. (2018) show that neural network architec- tures with multi-task learning are able to achieve comparable results to Anand and Hardt, without relying on structured syntactic annotation or hand- crafted features. On a slightly different version of the news dataset, Rønning et al. report a token- level F1 score of 0.70, compared to 0.67 for Anand and Hardt’s system. Furthermore, it is far superior to Anand and Hardt’s system at adapting from the newswire to a dialogue dataset. This is quite surprising as sluicing is tradition- ally understood to be constituent-based. Two ex- planations present themselves; first, the traditional view might simply be wrong – that is, linguistic structure is not actually needed for ellipsis resolu- tion. The second, and perhaps more reasonable, explanation is that Rønning et al.’s multi-task neu- ral network architectures have learned to extract and incorporate the relevant linguistic representa- tions. In this paper, we investigate the linguistic knowledge learned implicitly in the experiments in Rønning et al. (2018). We take two approaches to this: 1. We select linguistically-defined subsets of the data, and examine the output of different sys- tems on these subsets; and 2. we examine activations of the networks, fo- cusing in particular on the activations associ- ated with the wh-word that identifies a sluice, to assess how well the network notices, re- members, and classifies them. 2 Systems The two sluicing …

Content truncated to 500 words