Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results

Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results Matt Crane David R. Cheriton School of Computer Science, University of Waterloo matt.crane@uwaterloo.ca Abstract “Based on theoretical reasoning it has been suggested that the reliability of findings pub- lished in the scientific literature decreases with the popularity of a research field” (Pfeiffer and Hoffmann, 2009). As we know, deep learning is very popular and the ability to reproduce re- sults is an important part of science. There is growing concern within the deep learning community about the reproducibility of results that are presented. In this paper we present a number of controllable, yet unreported, ef- fects that can substantially change the effec- tiveness of a sample model, and thusly the re- producibility of those results. Through these environmental effects we show that the com- monly held belief that distribution of source code is all that is needed for reproducibility is not enough. Source code without a repro- ducible environment does not mean anything at all. In addition the range of results produced from these effects can be larger than the ma- jority of incremental improvement reported. 1 Introduction The recent “reproducibility crisis” (Baker, 2016) in various scientific fields (particularly Psychology and Social Sciences) indicates that some introspection is needed in all fields, particularly those that are exper- imental by nature. The efforts of Collberg’s repeata- bility studies highlight the state of affairs within the computer systems research community (Moraila et al., 2014; Collberg et al., 2015).1 Other fields have also begun to push for more stringent presentation of 1http://reproducibility.cs.arizona.edu results, for example, the information retrieval com- munity has been aware for some time of the issues surrounding weak baselines (Armstrong et al., 2009) and more recently reproducibility (Arguello et al., 2016; Lin et al., 2016). The issue of reproducibility in the deep-learning community has also started to become a growing concern, with the need for replicable and repro- ducible results being included in a list of challenges for the ACL (Nivre, 2017). In reinforcement learn- ing, Henderson et al. (2017) showed that there are a number of effects that would change the results ob- tained by published authors and call for more rigor- ous testing, and reporting, of state-of-the-art meth- ods. There is also an ongoing project by OpenAI to provide baselines in reinforcement learning that are reproduced from published descriptions, but even they admit that their scores are only “roughly on par with the scores in published papers.”2 Reimers and Gurevych (2017) investigated over 50,000 combina- tions of hyper-parameter settings, such as word em- bedding sources and the optimizer across five dif- ferent NLP tasks and found that these settings have a significant impact on both the variability, and the relative effectiveness of models. In this paper we present a number of controllable environment settings that often go unreported, and illustrate that these are factors that can cause irre- producibility of results as presented in the litera- ture. These environmental factors have an effect on the effectiveness …

Content truncated to 500 words