Mask-Align: Self-Supervised Neural Word Alignment

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 428–442 August 1–6, 2021. ©2021 Association for Computational Linguistics 428 Introducing Orthogonal Constraint in Structural Probes Tomasz Limisiewicz and David Mareˇcek Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics Charles University, Prague, Czech Republic {limisiewicz, marecek}@ufal.mff.cuni.cz Abstract With the recent success of pre-trained mod- els in NLP, a significant focus was put on in- terpreting their representations. One of the most prominent approaches is structural prob- ing (Hewitt and Manning, 2019), where a linear projection of word embeddings is per- formed in order to approximate the topology of dependency structures. In this work, we intro- duce a new type of structural probing, where the linear projection is decomposed into 1. iso- morphic space rotation; 2. linear scaling that identifies and scales the most relevant dimen- sions. In addition to syntactic dependency, we evaluate our method on novel tasks (lexical hypernymy and position in a sentence). We jointly train the probes for multiple tasks and experimentally show that lexical and syntactic information is separated in the representations. Moreover, the orthogonal constraint makes the Structural Probes less vulnerable to memoriza- tion. 1 Introduction Latent representations of neural networks encode specific linguistic features. Recently, a lot of fo- cus was devoted to interpret these representations and analyze structures captured by the deep mod- els. One of the most popular analysis methods is probing (Belinkov et al., 2017; Blevins et al., 2018; Linzen et al., 2016; Liu et al., 2019). The pre-trained model’s 1 parameters are fixed, and its latent states or outputs are then fed into a simple neural network optimized to solve an auxiliary task, e.g., semantic, syntactic parsing, anaphora resolu- tion, morphosyntactic tagging, etc. The amount of language information stored in the representa- tions can be evaluated by measuring the specific language task’s performance. 1Typically models for language modeling or machine trans- lation are analyzed. (a) Structural Probe (b) Orthogonal Structural Probe Figure 1: Comparison of the Structural Probe of He- witt and Manning (2019) and the Orthogonal Struc- tural Probe proposed by us. Probing experiments usually involve classifica- tion tasks. Lately, Hewitt and Manning (2019) proposed Structural Probes, which use regression as an optimization objective. They train a linear projection layer to approximate: 1. dependency tree distances between words2 by the Euclidean distance between transformed vectors; 2. the tree depth of a word by the norm of its vector. In Figure 1, we visualize our Orthogonal Struc- tural Probe. A linear transformation is replaced by an Orthogonal Transformation (rotation of the em- bedding space), and product-wise multiplication of rotated vectors by a Scaling Vector to get the final projections. Our motivation is to obtain an embed- ding space that is isomorphic with the original one, and the impact of each dimension can be evaluated 2Tree distance is the length of the tree path between two tokens 429 by analyzing Scaling Vector’s weights. We elabo- …

Content truncated to 500 words