PuzzLing Machines: A Challenge on Learning From Small Data

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1241–1254 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1241 PuzzLing Machines: A Challenge on Learning From Small Data G¨ozde G¨ul S¸ahin†, Yova Kementchedjhievaα, Phillip Rust†, Iryna Gurevych† †Ubiquitous Knowledge Processing Lab (UKP), Department of Computer Science, Technical University of Darmstadt αDepartment of Computer Science, University of Copenhagen †www.ukp.tu-darmstadt.de Abstract Deep neural models have repeatedly proved excellent at memorizing surface patterns from large datasets for various ML and NLP bench- marks. They struggle to achieve human-like thinking, however, because they lack the skill of iterative reasoning upon knowledge. To ex- pose this problem in a new light, we intro- duce a challenge on learning from small data, PuzzLing Machines, which consists of Rosetta Stone puzzles from Linguistic Olympiads for high school students. These puzzles are care- fully designed to contain only the minimal amount of parallel text necessary to deduce the form of unseen expressions. Solving them does not require external information (e.g., knowledge bases, visual signals) or linguis- tic expertise, but meta-linguistic awareness and deductive skills. Our challenge contains around 100 puzzles covering a wide range of linguistic phenomena from 81 languages. We show that both simple statistical algo- rithms and state-of-the-art deep neural mod- els perform inadequately on this challenge, as expected. We hope that this benchmark, available at https://ukplab.github.io/ PuzzLing-Machines/, inspires further ef- forts towards a new paradigm in NLP—one that is grounded in human-like reasoning and understanding. 1 Introduction Kahneman (2011) discusses the two modes of hu- man thinking which perfectly encapsulate the cur- rent (so called System1) and the desired state (Sys- tem1+System2) of the deep learning field. System1 handles tasks that humans consider fast, intuitive and automatic, such as object detection and doc- ument classification. Recent deep learning (DL) models have shown great promise at this type of tasks—thanks to large training datasets. Yet, it is through slow, rational and sequential mecha- nisms that human-like abstract reasoning happens, Chikasaw English 1. Ofi’at kowi’˜a lhiyohli. The dog chases the cat. 2. Kowi’at ofi’˜a lhiyohli. The cat chases the dog. 3. Ofi’at shoha. The dog stinks. 4. Ihooat hattak˜a hollo. The woman loves the man. 5. Lhiyohlili. I chase her/him. 6. Salhiyohli. She/he chases me. 7. Hilha. She/he dances. Now you can translate the following into Chickasaw: The man loves the woman. The cat stinks. I love her/him. Translate the following into English: Ihooat sahollo. Ofi’at hilha. Kowi’˜a lhiyohlili. Table 1: The “Chickasaw” puzzle (Payne, 2005) to enable learning from just a few examples. This System2-style modeling is still in its early stages in DL, but is recognized as a much needed next step in the field (McClelland et al., 2019; Marcus, 2020; LeCun, 2020; Bengio, 2020). To foster research in this promising direction, we propose a unique chal- lenge on “learning from small data”: PuzzLing Ma- chines, based on the Linguistic Olympiads—one of the 13 recognized International Science Olympiads targeted at high-school students. The PuzzLing Machines challenge is based on …

Content truncated to 500 words