Discriminatively-Tuned Generative Classifiers for Robust Natural Language Inference

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8189–8202, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8189 Discriminatively-Tuned Generative Classifiers for Robust Natural Language Inference Xiaoan Ding1 ∗ Tianyu Liu3 ∗ † Baobao Chang3 4 Zhifang Sui3 4 Kevin Gimpel2 1 University of Chicago, IL, USA 2 Toyota Technological Institute at Chicago, IL, USA 3 Peking University, Beijing, China 4 Peng Cheng Laboratory, Shenzhen, China xiaoanding@uchicago.edu, {tianyu0421, chbb, szf}@pku.edu.cn, kgimpel@ttic.edu Abstract While discriminative neural network classi- fiers are generally preferred, recent work has shown advantages of generative classifiers in term of data efficiency and robustness. In this paper, we focus on natural language in- ference (NLI). We propose GenNLI, a gen- erative classifier for NLI tasks, and empiri- cally characterize its performance by compar- ing it to five baselines, including discrimi- native models and large-scale pretrained lan- guage representation models like BERT. We explore training objectives for discriminative fine-tuning of our generative classifiers, show- ing improvements over log loss fine-tuning from prior work (Lewis and Fan, 2019). In particular, we find strong results with a sim- ple unbounded modification to log loss, which we call the “infinilog loss”. Our experiments show that GenNLI outperforms both discrimi- native and pretrained baselines across several challenging NLI experimental settings, includ- ing small training sets, imbalanced label distri- butions, and label noise. 1 Introduction Natural language inference (NLI) is the task of identifying the relationship between two fragments of text, called the premise and the hypothesis (Da- gan et al., 2005; Dagan et al., 2013). The task was originally defined as binary classification, in which the labels are entailment (the premise im- plies the hypothesis) or not entailment. Subsequent variations added a third contradiction label. Most models for NLI are trained and evaluated on stan- dard benchmarks (Bowman et al., 2015; Williams et al., 2018; Wang et al., 2018) in a discriminative manner (Conneau et al., 2017; Chen et al., 2017a). These benchmarks typically have relatively clean, balanced, and abundant annotated data, and there ∗Equal contribution. †Contribution during visiting TTIC. is no distribution shift between the training and test sets. However, when data quality and conditions are not ideal, there is a substantial performance de- crease for existing discriminative models, includ- ing both simple model architectures and more com- plex ones. Prior work on document classification and question answering has shown that genera- tive classifiers have advantages over their discrim- inative counterparts in non-ideal conditions (Yo- gatama et al., 2017; Lewis and Fan, 2019; Ding and Gimpel, 2019). In this paper, we develop generative classifiers for NLI. Our model, which we call GenNLI, de- fines the conditional probability of the hypothesis given the premise and the label, parameterizing the distribution using a sequence-to-sequence model with attention (Luong et al., 2015) and a copy mechanism (Gu et al., 2016). We explore train- ing objectives for discriminative fine-tuning of our generative classifiers, comparing several classical discriminative criteria. We find that several losses, including hinge loss and softmax-margin, outper- form log …

Content truncated to 500 words