Halo: Learning Semantics-Aware Representations for Cross-Lingual Information Extraction

Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), pages 142–147 New Orleans, June 5-6, 2018. c⃝2018 Association for Computational Linguistics Halo: Learning Semantics-Aware Representations for Cross-Lingual Information Extraction Hongyuan Mei∗, Sheng Zhang∗, Kevin Duh, Benjamin Van Durme Center for Language and Speech Processing, Johns Hopkins University {hmei,s.zhang,kevinduh,vandurme}@cs.jhu.edu Abstract Cross-lingual information extraction (CLIE) is an important and challenging task, especially in low resource scenarios. To tackle this chal- lenge, we propose a training method, called Halo, which enforces the local region of each hidden state of a neural model to only gener- ate target tokens with the same semantic struc- ture tag. This simple but powerful technique enables a neural model to learn semantics- aware representations that are robust to noise, without introducing any extra parameter, thus yielding better generalization in both high and low resource settings. 1 Introduction Cross-lingual information extraction (CLIE) is the task of distilling and representing factual informa- tion in a target language from the textual input in a source language (Sudo et al., 2004; Zhang et al., 2017b). For example, Fig. 1 illustrates a pair of input Chinese sentence and its English predicate- argument information1, where predicate and argu- ment are well used semantic structure tags. It is of great importance to solve the task, as to provide viable solutions to extracting information from the text of languages that suffer from no or little existing information extraction tools. Neu- ral models have empirically proven successful in this task (Zhang et al., 2017b,c), but still remain unsatisfactory in low resource (i.e. small number of training samples) settings. These neural models learn to summarize a given source sentence and target prefix into a hidden state, which aims to generate the correct next target token after being ∗equal contribution 1The predicate-argument information is usually denoted by relation tuples. In this work, we adopt the tree-structured representation generated by PredPatt (White et al., 2016; Zhang et al., 2017d), which was a lightweight tool available at https://github.com/hltcoe/PredPatt. Figure 1: Example of cross-lingual information extraction: Chinese input text (a) and linearized English PredPatt output (b), where ‘:p’ and blue stand for predicate while ‘:a’ and purple denote argument. passed through an output layer. As each member in the target vocabulary is essentially either pred- icate or argument, a random perturbation on the hidden state should still be able to yield a token with the same semantic structure tag. This induc- tive bias motivates an extra term in training objec- tive, as shown in Fig. 2, which enforces the sur- roundings of any learned hidden state to generate tokens with the same semantic structure tag (either predicate or argument) as the centroid. We call this technique Halo, because the process of each hidden state taking up its surroundings is analo- gous to how the halo is formed around the sun. The method is believed to help the model gen- eralize better, by learning more semantics-aware and noise-insensitive hidden states without intro- ducing extra parameters. 2 The Problem We are …

Content truncated to 500 words