It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning

It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning Alexey Tikhonov∗ Yandex Berlin, Germany altsoph@gmail.com Max Ryabinin∗ Yandex, HSE University Moscow, Russia mryabinin0@gmail.com Abstract Commonsense reasoning is one of the key problems in natural language processing, but the relative scarcity of labeled data holds back the progress for languages other than English. Pretrained cross-lingual models are a source of powerful language-agnostic representations, yet their inherent reasoning capabilities are still actively studied. In this work, we design a simple approach to commonsense reasoning which trains a linear classifier with weights of multi-head attention as features. To eval- uate this approach, we create a multilingual Winograd Schema corpus by processing sev- eral datasets from prior work within a standard- ized pipeline and measure cross-lingual gen- eralization ability in terms of out-of-sample performance. The method performs competi- tively with recent supervised and unsupervised approaches for commonsense reasoning, even when applied to other languages in a zero-shot manner. Also, we demonstrate that most of the performance is given by the same small sub- set of attention heads for all studied languages, which provides evidence of universal reason- ing capabilities in multilingual encoders. 1 Introduction Neural networks have achieved remarkable progress in numerous tasks involving natural lan- guage, such as machine translation (Bahdanau et al., 2014; Kaplan et al., 2020; Arivazhagan et al., 2019), language modeling (Brown et al., 2020), open-domain dialog systems (Adiwardana et al., 2020; Roller et al., 2020), and general-purpose language understanding (Devlin et al., 2019; He et al., 2021). However, the fundamental problem of commonsense reasoning has proven to be quite challenging for modern methods and arguably re- mains unsolved up to this day. The tasks that aim to ∗Equal contribution. The town councilors refused to give the demonstra- tors a permit because they feared violence. Answer: The town councilors Figure 1: Example of a Winograd Schema problem. The resolved pronoun is underlined, two options are highlighted with an italic font. measure reasoning capabilities, such as the Wino- grad Schema Challenge (Levesque et al., 2012), are deliberately designed not to be easily solved by statistical approaches, which are a foundation of most deep learning methods. Instead, these tasks require implicit knowledge about properties of real- world entities and their relations in order to resolve inherent ambiguities of natural language. Figure 1 illustrates the gist of this task: given a sentence and a pronoun (they), the goal is to choose the word that this pronoun refers to from two op- tions (The town councilors or the demonstrators). While picking the right answer is straightforward for humans, the lack of explicit clues makes it hard for machine learning algorithms to perform better than majority vote or random choice. Recently large Transformer-based masked lan- guage models (MLMs) (Devlin et al., 2019) were shown to achieve impressive results on several benchmark datasets for commonsense reasoning (Sakaguchi et al., 2020; Kocijan et al., 2019; Klein and Nabi, 2020). However, the best-performing methods frequently involve finetuning the entire model …

Content truncated to 500 words