Look at the First Sentence: Position Bias in Question Answering

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1109–1121, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1109 Look at the First Sentence: Position Bias in Question Answering Miyoung Ko Jinhyuk Lee† Hyunjae Kim Gangwoo Kim Jaewoo Kang† Korea University {miyoungko,jinhyuk lee,hyunjae-kim}@korea.ac.kr {gangwoo kim,kangj}@korea.ac.kr Abstract Many extractive question answering models are trained to predict start and end positions of answers. The choice of predicting answers as positions is mainly due to its simplicity and ef- fectiveness. In this study, we hypothesize that when the distribution of the answer positions is highly skewed in the training set (e.g., answers lie only in the k-th sentence of each passage), QA models predicting answers as positions can learn spurious positional cues and fail to give answers in different positions. We first illustrate this position bias in popular extrac- tive QA models such as BiDAF and BERT and thoroughly examine how position bias propa- gates through each layer of BERT. To safely deliver position information without position bias, we train models with various de-biasing methods including entropy regularization and bias ensembling. Among them, we found that using the prior distribution of answer positions as a bias model is very effective at reducing position bias, recovering the performance of BERT from 37.48% to 81.64% when trained on a biased SQuAD dataset. 1 Introduction Question answering (QA) is a task of answering questions given a passage. Large-scale QA datasets have attracted many researchers to build effective QA models, and with the advent of deep learn- ing, recent QA models are known to outperform humans in some datasets (Rajpurkar et al., 2016; Devlin et al., 2019; Yang et al., 2019). Extractive QA is the task that assumes that answers always lie in the passage. Based on this task assumption, various QA models are trained to predict the start and end positions as the answers. Following the †Corresponding authors Question: When was the Royal University of Warsaw established? Answer: 1816 (kth sent.) Warsaw remained the capital of the Polish–Lithuanian …Commonwealth until 1796, (Last sent.) The Royal University of Warsaw was established in 1816. Prediction Test Sample … Training data (All answers are in the kth sentence) Model Prediction Answer Context (1st sent.) … (kth sent.) (k+1th sent.) … (Question, Answer) Example #2 Context (1st sent.) … (kth sent.) (k+1th sent.) … (Question, Answer) Example #1 Figure 1: Example of position bias. BERT trained on the dataset with a skewed answer position distribution, provides wrong predictions, biased to the specific sen- tence position. structure of earlier deep learning-based QA mod- els (Wang and Jiang, 2016; Seo et al., 2017; Xiong et al., 2017), recent QA models provide positions of answers without much consideration (Yu et al., 2018; Devlin et al., 2019; Yang et al., 2019). The popularity of predicting the answer posi- tions is credited to the fact that it reduces the pre- diction space to O(n) where n is the length of an input document. It is more efficient and effective than directly …

Content truncated to 500 words