Humans Keep It One Hundred: an Overview of AI Journey

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2276–2284 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2276 Humans Keep It One Hundred: an Overview of AI Journey Tatiana Shavrina1,3, Anton Emelyanov1,4, Alena Fenogenova1, Vadim Fomin1,3, Vladislav Mikhailov1,3, Andrey Evlampiev1, Valentin Malykh2, Vladimir Larin1, Alex Natekin6,7, Aleksandr Vatulin6,7, Peter Romov6,7, Daniil Anastasyev4,5, Nikolai Zinov4,5, Andrey Chertok1 1Sberbank, 2Huawei, 3NRU HSE, 4MIPT, 5Yandex, 6Data Souls, 7ODS.ai Moscow, Russia rybolos@gmail.com Abstract Artificial General Intelligence (AGI) is showing growing performance in numerous applications - beating human performance in Chess and Go, using knowledge bases and text sources to answer questions and even pass school student examination. In this paper, we describe the results of AI Journey, a competition of AI-systems aimed to improve AI performance on linguistic knowledge evaluation, reasoning and text generation. Competing systems have passed Unified State Exam (USE, in Russian), including versatile grammar tasks (test and open questions) and an essay: a combined solution consisting of the best performing models have achieved a high score of 69%, with 68% being an average human result. During the competition, a baseline for the task and essay parts was proposed, and 98 systems were submitted, showing different approaches to task solving and reasoning. All the data and solutions can be found on github https://github.com/sberbank-ai/combined_solution_aij2019 Keywords: artificial intelligence, agi, artificial general intelligence, Russian language, question answering 1. Introduction Since the Turing Test was introduced (Machinery, 1950), the number of different ways AI systems are assessed has significantly grown (Potthast et al., 2013; Bellemare et al., 2013; Caputo et al., 2014). Recently, the Robot College Student Test has been proposed to confirm human-level ar- tificial general intelligence (AGI) on the capability to enrol in a university and take exams in the same way humans do (Goertzel, 2014). The test requires advanced compre- hension of natural language (NLU), along with the capabil- ity to support reasoning, use of commonsense knowledge and answer generation. The AI Journey competition is de- signed to test AI systems on passing the Unified State Exam (USE) in the Russian language in full concordance with the official guidelines and knowledge assessment system, in- cluding an automatically evaluated knowledge testing and human-based evaluation of the essays. 2. Motivation Previously, a few exam-oriented question answering con- tests were organized in English (Clark, 2015), Chinese (Cheng et al., 2016; Guo et al., 2017), and Japanese (Strick- land, 2013; Fujita et al., 2014) languages. At the most large-scale among all such contests, Allen AI Science Chal- lenge (Clark, 2015), systems were designed to answer stan- dard 8th grade multiple choice science questions. More so- phisticated AI knowledge and reasoning abilities were as- sessed in the AI2 Reasoning Challenge (ARC) as described in (Clark et al., 2018). The Aristo system (Clark et al., 2019) has achieved remarkable success on the Grade 8 New York Regents Science Exam covering more than 90% of the exam’s multiple choice and non-diagram questions, and scored 83% on the Grade 12 Science …

Content truncated to 500 words