Evaluating Interactive System Adaptation

Evaluating Interactive System Adaptation Edouard Geoffrois Agence Nationale de la Recherche (ANR)∗ 50 avenue Daumesnil, 75012 Paris, France edouard.geoffrois@anr.fr Abstract Enabling users of intelligent systems to enhance the system performance by providing feedback on their errors is an important need. However, the ability of systems to learn from user feedback is difficult to evaluate in an objective and comparative way. Indeed, the involvement of real users in the adaptation process is an impediment to objective evaluation. This issue can be solved by using an oracle approach, where users are simulated by oracles having access to the reference test data. Another difficulty is to find a meaningful metric despite the fact that system improvements depend on the feedback provided and on the system itself. A solution is to measure the minimal amount of information needed to correct all system errors. It can be shown that for any well defined non interactive task, the interactively supervised version of the task can be evaluated by combining such an oracle-based approach and a minimum supervision rate metric. This new evaluation protocol for adaptive systems is not only expected to drive progress for such systems, but also to pave the way for a specialisation of actors along the value chain of their technological development. Keywords: Evaluation methodologies; System adaptation; Interactive systems 1. Introduction Interactive system adaptation, i.e., the ability of a system to learn from user feedback, is an important need for many ap- plications. For example, in the domain of automatic speech recognition, a simple need is to enable users to provide new words in order to have the system recognize them. This need has been identified long ago (Asadi et al., 1991) and is still a topic of active research (Orosanu and Jouvet, 2015). Another example is computer-assisted translation, where the user edits an automatic translation (Isabelle and Church, 1997) and can expect to produce a correct trans- lation more efficiently if the system learns from this feed- back. Still another one is interactive information retrieval, where the user provides feedback on the relevance of re- trieved documents to improve the overall search results, be it for textual (Salton and Buckley, 1990) or multimedia doc- uments (Nguyen and Worring, 2008). However, in general, evaluating the ability of systems to learn from user feedback in an objective and comparative way is difficult. Indeed, since human users are in the loop of the evaluation, such systems are often evaluated in a subjec- tive way, which is expensive and not exactly reproducible. Another difficulty to compare approaches is that the im- provements in performance obtained through the adapta- tion depend on the amount of information available for this adaptation, and in an iterative setting this amount is not un- der the control of the evaluator but of the user. In practice, an objective evaluation protocol has been in- troduced only in some specific cases, such as interactive- predictive machine translation, which is a special case of *The author is currently on loan from the French national …

Content truncated to 500 words