Don’t take “nswvtnvakgxpm” for an answer –The surprising vulnerability of automatic content scoring systems to adversarial input

Proceedings of the 28th International Conference on Computational Linguistics, pages 882–892 Barcelona, Spain (Online), December 8-13, 2020 882 Don’t take “nswvtnvakgxpm” for an answer – The surprising vulnerability of automatic content scoring systems to adversarial input Yuning Ding1 Brian Riordan2 Andrea Horbach1 1Language Technology Lab, University of Duisburg-Essen, Duisburg, Germany 2Educational Testing Service, Princeton, NJ 08541, USA Aoife Cahill2 Torsten Zesch1 Abstract Automatic content scoring systems are widely used on short answer tasks to save human effort. However, the use of these systems can invite cheating strategies, such as students writing irrelevant answers in the hopes of gaining at least partial credit. We generate adversarial answers for benchmark content scoring datasets based on different methods of increasing sophistication and show that even simple methods lead to a surprising decrease in content scoring performance. As an extreme example, up to 60% of adversarial answers generated from random shuffling of words in real answers are accepted by a state-of-the-art scoring system. In addition to analyzing the vulnerabilities of content scoring systems, we examine countermeasures such as adversarial training and show that these measures improve system robustness against adversarial answers considerably but do not suffice to completely solve the problem. 1 Introduction Automatic content scoring (Ziai et al., 2012) is an educational task where free-text answers to a content question are automatically scored based on the conceptual correctness of the answers. Consider for example the question What is the favorite food of pandas? with an expected answer such as Pandas eat bamboo. A content scoring system which only accepted this exact same answer would be very limited, as there are many other ways to express the same fact, e.g. Pandas like to eat bamboo, Pandas feed on bamboo, or simply bamboo. In addition, students might make all kinds of spelling and grammatical mistakes, which are usually ignored in content scoring as long as they do not interfere with understandability of the answer. Thus, Pendas are eating bambboo would still be acceptable. A content scoring system must therefore be able to generalize beyond a certain set of a prior known correct answers, e.g. by taking variance in spelling (Leacock and Chodorow, 2003) and from other sources into account (Horbach and Zesch, 2019). For example, many supervised scoring systems rely on character-level n-gram features (Heilman and Madnani, 2013; Zesch et al., 2015). Without those features, the system would not be able to detect that bambboo is likely to mean bamboo as they both e.g. contain the trigrams ‘bam’ and ‘amb’. In fact, such systems show remarkable resilience against spelling errors, where even introducing up to 50% spelling errors in each answer does not significantly influence scoring performance (Horbach et al., 2017). This ability of a system to generalize beyond previously seen examples makes it potentially vulnerable to adversarial behaviour. In the educational domain such behaviour has long been known under the term cheating. When students have no idea about a task, they could, for example, try to write an answer combining several potential answers, such …

Content truncated to 500 words