Software Testing and the Naturally Occurring Data Assumption in Natural Language Processing

Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pages 23–30, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Software testing and the naturally occurring data assumption in natural language processing∗ K. Bretonnel Cohen William A. Baumgartner, Jr. Lawrence Hunter Abstract It is a widely accepted belief in natural lan- guage processing research that naturally oc- curring data is the best (and perhaps the only appropriate) data for testing text mining sys- tems. This paper compares code coverage us- ing a suite of functional tests and using a large corpus and finds that higher class, line, and branch coverage is achieved with structured tests than with even a very large corpus. 1 Introduction In 2006, Geoffrey Chang was a star of the protein crystallography world. That year, a crucial compo- nent of his code was discovered to have a simple error with large consequences for his research. The nature of the bug was to change the signs (positive versus negative) of two columns of the output. The effect of this was to reverse the predicted “handed- ness” of the structure of the molecule—an impor- tant feature in predicting its interactions with other molecules. The protein for his work on which Chang was best known is an important one in predicting things like human response to anticancer drugs and the likelihood of bacteria developing antibiotic re- sistance, so his work was quite influential and heav- ily cited. The consequences for Chang were the withdrawal of 5 papers in some of the most presti- gious journals in the world. The consequences for the rest of the scientific community have not been ∗K. Bretonnel Cohen is with The MITRE Corporation. All three co-authors are at the Center for Computational Pharma- cology in the University of Colorado School of Medicine. quantified, but were substantial: prior to the retrac- tions, publishing papers with results that did not jibe with his model’s predictions was difficult, and obtaining grants based on preliminary results that seemed to contradict his published results was dif- ficult as well. The Chang story (for a succinct dis- cussion, see (Miller, 2006), and see (Chang et al., 2006) for the retractions) is an object illustration of the truth of Rob Knight’s observation that “For sci- entific work, bugs don’t just mean unhappy users who you’ll never actually meet: they mean retracted publications and ended careers. It is critical that your code be fully tested before you draw conclu- sions from results it produces” (personal communi- cation). Nonetheless, the subject of software testing has been largely neglected in academic natural lan- guage processing. This paper addresses one aspect of software testing: the monitoring of testing efforts via code coverage. 1.1 Code coverage Code coverage is a numerical assessment of the amount of code that is executed during the running of a test suite (McConnell, 2004). Although it is by no means a completely sufficient method for de- termining the completeness of a testing effort, it is nonetheless a helpful member of …

Content truncated to 500 words