Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment

Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment Masatoshi Tsuchiya Toyohashi University of Technology 1–1 Hibarigaoka, Tempaku-cho, Toyohashi, Aichi, Japan tsuchiya@imc.tut.ac.jp Abstract The quality of training data is one of the crucial problems when a learning-centered approach is employed. This paper proposes a new method to investigate the quality of a large corpus designed for the recognizing textual entailment (RTE) task. The proposed method, which is inspired by a statistical hypothesis test, consists of two phases: the first phase is to introduce the predictability of textual entailment labels as a null hypothesis which is extremely unacceptable if a target corpus has no hidden bias, and the second phase is to test the null hypothesis using a Naive Bayes model. The experimental result of the Stanford Natural Language Inference (SNLI) corpus does not reject the null hypothesis. Therefore, it indicates that the SNLI corpus has a hidden bias which allows prediction of textual entailment labels from hypothesis sentences even if no context information is given by a premise sentence. This paper also presents the performance impact of NN models for RTE caused by this hidden bias. Keywords: Recognizing Textual Entailment; Evaluation Methodology; Naive Bayes 1. Introduction The quality of a training data is one of the crucial prob- lems when a learning-centered approach including neural network (NN) is employed. (Reidsma and Carletta, 2008) demonstrated that annotation errors of the dialog act cor- pus, which follows a certain systematic pattern, mislead the learning result of Bayesian network. (Zhang et al., 2017) described that the capacity of a NN model is large enough for brute-force memorizing the entire data set, even if its la- bels are random. Thus, an influence for a NN model caused by a certain systematic pattern may become more serious than the influence for other learning-centered methods. Both a method to improve the quality of a training data and a metric to evaluate its reliability are important. As the former method, majority vote of human annotators was widely used (Sabou et al., 2014; Bowman et al., 2015; Al Khatib et al., 2016) As the latter metric, many kind of inter-annotator agreement metrics based on multiple human annotation results were employed, because direct assess- ment of data quality is difficult (Craggs and Wood, 2005; Artstein and Poesio, 2008; Mathet et al., 2012). This paper proposes a new empirical method to investi- gate a quality of a large corpus designed for the recogniz- ing textual entailment (RTE) task (Condoravdi et al., 2003; Bos and Markert, 2005; MacCartney and Manning, 2009; Marelli et al., 2014a). The proposed method, which is in- spired by a statistical hypothesis test, assesses a quality of a target corpus directly and does not depend on multiple human annotation results unlike the existing metrics. The proposed method consists of two phases: the first phase is to introduce the predictability of textual entailment (TE) la- bels as a null hypothesis which is extremely unacceptable when a target corpus has no hidden bias, and the …

Content truncated to 500 words