Evaluation of Lifelong Learning Systems

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1833–1841 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1833 Evaluation of Lifelong Learning Systems Yevhenii Prokopalo1, Sylvain Meignier1, Olivier Galibert2 Lo¨ıc Barrault3, Anthony Larcher1 1. LIUM, Le Mans University, 2. LNE, 3. University of Sheffield 1firstname.lastname@univ-lemans.fr, 2olivier.galibert@lne.fr,3l.barrault@sheffield.ac.uk Abstract Current intelligent systems need the expensive support of machine learning experts to sustain their performance level when used on a daily basis. To reduce this cost, i.e. remaining free from any machine learning expert, it is reasonable to implement lifelong (or continuous) learning intelligent systems that will continuously adapt their model when facing changing execution conditions. In this work, the systems are allowed to refer to human domain experts who can provide the system with relevant knowledge about the task. Nowadays, the fast growth of lifelong learning systems development rises the question of their evaluation. In this article we propose a generic evaluation methodology for the specific case of lifelong learning systems. Two steps will be considered. First, the evaluation of human-assisted learning (including active and/or interactive learning) outside the context of lifelong learning. Second, the system evaluation across time, with propositions of how a lifelong learning intelligent system should be evaluated when including human assisted learning or not. Keywords: Evaluation, lifelong learning, human assisted learning 1. Introduction Today’s intelligent systems are used in many fields to pro- cess data for various tasks amongst which classification and regression (Prince, 2012; Bishop, 2006). Intelligent systems make use of a representation of the world around them, a model, that is initially learnt in laboratories un- der supervision of machine learning (ML) experts whose role is threefold. First, they select and label the so-called training data required to learn the model used by the sys- tem. One can expect the ML experts to select training data that is representative of what the system will be exposed to in the real world in a near future. Second, they set the numerous meta-parameters inherent to the system architec- ture and train the model by using development data in order to define the meta-parameters of the system and their opti- mal value. Third, before releasing it in the real world they benchmark the performance of the system on a third data set referred to as test data that is used to assert the general- isation of the system performance (in order to avoid over- fitting). In this paper we will use the term ”initialisation data” to refer to the set of training, development and test data together. Once in the real world, intelligent systems are exposed to new incoming data. The life-cycle of those standard auto- matic systems, which will be referred to as static learning or isolated learning systems (Chen and Liu, 2016), is de- scribed in Figure 1-a. At first, one can expect incoming data to be similar to the initialisation data but, across time, the distribution of this data might move away from the one of …

Content truncated to 500 words