Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4963–4969, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4963 Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution Yordan Yordanov1, Oana-Maria Camburu1,2, Vid Kocijan1, Thomas Lukasiewicz1,2 1University of Oxford, Oxford, UK 2Alan Turing Institute, London, UK firstname.lastname@cs.ox.ac.uk Abstract Hard cases of pronoun resolution have been used as a long-standing benchmark for com- monsense reasoning. In the recent literature, pre-trained language models have been used to obtain state-of-the-art results on pronoun resolution. Overall, four categories of train- ing and evaluation objectives have been intro- duced. The variety of training datasets and pre- trained language models used in these works makes it unclear whether the choice of train- ing objective is critical. In this work, we make a fair comparison of the performance and seed- wise stability of four models that represent the four categories of objectives. Our experiments show that the objective of sequence ranking performs the best in-domain, while the objec- tive of semantic similarity between candidates and pronoun performs the best out-of-domain. We also observe a seed-wise instability of the model using sequence ranking, which is not the case when the other objectives are used. 1 Introduction Hard cases of pronoun resolution have been a long- standing problem in natural language processing, which has served as a performance benchmark for the research community (Levesque et al., 2012; Wang et al., 2018, 2019a). For example, the Wino- Grande dataset (Sakaguchi et al., 2019) consists of pronoun resolution schemas that are constructed so that resolving them requires background knowl- edge and commonsense reasoning. In WinoGrande, the pronoun is obscured by “ ” to remove gender and number cues. The task is to find the correct candidate for “ ” out of two given candidates. For example: John moved the couch from the garage to the backyard to create space. The is small. Candidates: garage, backyard. Recently, supervised learning on top of pre- trained language models has been established as the main approach for pronoun resolution (Kocijan et al., 2019b,a; Sakaguchi et al., 2019). Under this type of approach, we identify four categories of objectives commonly used for pronoun resolution: 1. comparing the language model probabilities for each candidate (Kocijan et al., 2019b,a; He et al., 2019), 2. using semantic similarity between the pro- noun and the candidates (Wang et al., 2019b; He et al., 2019), 3. using sequence ranking among the possible substituted sentences (Opitz and Frank, 2018; Sakaguchi et al., 2019), and 4. selecting a candidate based on the attentions of the pronoun in a transformer model (Klein and Nabi, 2019). We list one representative model from each cate- gory. For 1, Kocijan et al. (2019b) use the BERT masked language model (Devlin et al., 2018) to produce the probabilities of the pronoun to be re- placed with each of the two candidates. For 2, the Unsupervised Deep Structured Semantic Model (UDSSM-I) (Wang et al., 2019b) uses contextu- alized word embeddings produced by a bidirec- …

Content truncated to 500 words