Learning to Rank for Plausible Plausibility

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4818–4823 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4818 Learning to Rank for Plausible Plausibility Zhongyang Li†‡∗ Tongfei Chen‡ Benjamin Van Durme‡ † Harbin Institute of Technology ‡ Johns Hopkins University zyli@ir.hit.edu.cn, {tongfei,vandurme}@cs.jhu.edu Abstract Researchers illustrate improvements in contex- tual encoding strategies via resultant perfor- mance on a battery of shared Natural Lan- guage Understanding (NLU) tasks. Many of these tasks are of a categorical prediction vari- ety: given a conditioning context (e.g., an NLI premise), provide a label based on an associ- ated prompt (e.g., an NLI hypothesis). The cat- egorical nature of these tasks has led to com- mon use of a cross entropy log-loss objective during training. We suggest this loss is in- tuitively wrong when applied to plausibility tasks, where the prompt by design is neither categorically entailed nor contradictory given the context. Log-loss naturally drives models to assign scores near 0.0 or 1.0, in contrast to our proposed use of a margin-based loss. Fol- lowing a discussion of our intuition, we de- scribe a confirmation study based on an ex- treme, synthetically curated task derived from MultiNLI. We find that a margin-based loss leads to a more plausible model of plausibil- ity. Finally, we illustrate improvements on the Choice Of Plausible Alternative (COPA) task through this change in loss. 1 Introduction Contextualized encoders such as GPT (Radford et al., 2018) and BERT (Devlin et al., 2019) have led to improvements on various structurally simi- lar Natural Language Understanding (NLU) tasks such as variants of Natural Language Inference (NLI). Such tasks model the conditional interpreta- tion of a sentence (e.g., an NLI hypothesis) based on some other context (usually some other sentence, e.g., an NLI premise). The structural similarity of these tasks points to a structurally similar modeling approach: (1) concatenate the conditioning con- text (premise) to a sentence to be interpreted, (2) ∗This work was done while the first author was visiting Johns Hopkins University. p I just stopped where I was hE I stopped in my tracks ✓ hN I stopped running right were I was hN I stopped running right were I was ✓ hC I continued on my way Figure 1: COPA-like pairs may be constructed from datasets such as MultiNLI, where a premise and two hy- potheses are presented, where the correct – most plau- sible – item depends on the competing hypothesis. Score Distribution Cross entropy log-loss CON NEU ENT Score Distribution Margin-loss CON NEU ENT Figure 2: Dev set score distribution on COPA-pairs de- rived from MNLI, after training with cross entropy log- loss and margin-loss. Margin-loss leads to a more intu- itively plausible encoding of Neutral statements. read this pair using a contextualized encoder, then (3) employ the resultant representation to support classification under the label set of the task. NLI datasets employ a categorical label scheme (Entail- ment, Neutral, Contradiction) which has led to the use of a …

Content truncated to 500 words