Multiˆ2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1107–1117 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1107 Multi2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT Youngbin Ro Yukyung Lee Pilsung Kang† Korea University, Seoul, Republic of Korea {youngbin ro, yukyung lee, pilsung kang}@korea.ac.kr Abstract In this paper, we propose Multi2OIE, which performs open information extraction (open IE) by combining BERT (Devlin et al., 2019) with multi-head attention blocks (Vaswani et al., 2017). Our model is a sequence-labeling system with an efficient and effective argu- ment extraction method. We use a query, key, and value setting inspired by the Multi- modal Transformer (Tsai et al., 2019) to re- place the previously used bidirectional long short-term memory architecture with multi- head attention. Multi2OIE outperforms exist- ing sequence-labeling systems with high com- putational efficiency on two benchmark eval- uation datasets, Re-OIE2016 and CaRB. Addi- tionally, we apply the proposed method to mul- tilingual open IE using multilingual BERT. Ex- perimental results on new benchmark datasets introduced for two languages (Spanish and Portuguese) demonstrate that our model out- performs other multilingual systems without training data for the target languages. 1 Introduction Open information extraction (Open IE) (Banko et al., 2007) aims to extract a set of arguments and their corresponding relationship phrases from natu- ral language text. For example, an open IE system could derive the relational tuple (was elected; The Republican candidate; President) from the given sentence “The Republican candidate was elected President.” Because the extractions generated by open IE are considered as useful intermediate repre- sentations of the source text (Mausam, 2016), this method has been applied to various downstream tasks (Christensen et al., 2013; Ding et al., 2016; Khot et al., 2017; Wu et al., 2018). Although early open IE systems were largely based on handcrafted features or fine-grained rules †Corresponding author 𝐏���𝐔𝐃���𝐑𝐘 𝐊𝐃���𝐘 𝐔���𝐀𝐊���𝐔𝐃��� 𝑤1 𝑤2 𝑤3 … 𝑤𝑙−1 𝑤𝑙 [ 𝐃���𝐱𝐠���𝐫���𝐫���𝐠���𝐦���𝐠 𝐊���𝐚���𝐫���𝐠���𝐦���𝐚���𝐫��� ] [ 𝐏𝐫𝐦���𝐦���𝐦���𝐫���𝐚���𝐚��� 𝐊���𝐚���𝐫���𝐠���𝐦���𝐚��� ] 𝐏𝐫𝐚���𝐚���𝐠���𝐚���𝐚𝐫���𝐚��� 𝐃���𝐱𝐫���𝐫𝐚𝐚���𝐫���𝐠���𝐦���𝐦��� 𝐀𝐫𝐠𝐫���𝐦𝐚���𝐦���𝐫��� 𝐃���𝐱𝐫���𝐫𝐚𝐚���𝐫���𝐠���𝐦���𝐦��� 𝐃���𝐚���𝐚𝐫���𝐫���𝐫𝐚��� 𝐃���𝐦𝐚���𝐚���𝐚���𝐚���𝐠���𝐦���𝐠 Embedding Layers 𝐀𝐑𝐃���𝟎 𝐀𝐑𝐃���𝟎 𝐀𝐑𝐃���𝟎��� 𝐏𝐑𝐃���𝐃 Multi−Head Attention Block Embedding Layers tokens tokens 𝐀𝐑𝐃���𝟎 𝐀𝐑𝐃���𝟎 𝐀𝐑𝐃���𝟎��� 𝐏𝐑𝐃���𝐃 𝑤1 𝑤2 𝑤3 … 𝑤𝑙−1 𝑤𝑙 𝑤1 𝑤2 𝑤3 … 𝑤𝑙−1 𝑤𝑙 𝑒1 𝑒2 𝑒3 … 𝑒𝑙−1 𝑒𝑙 ℎ1 ℎ2 ℎ3 ℎ𝑙 … Bidirectional−LSTM Bidirectional−LSTM Figure 1: Comparison between existing extractors and the proposed method. We use BERT for feature embed- ding layers and as a predicate extractor. Predicate infor- mation is reflected through multi-head attention instead of simple concatenation. (Fader et al., 2011; Mausam et al., 2012; Del Corro and Gemulla, 2013), most recent open IE research has focused on deep-neural-network-based super- vised learning models. Such systems are typically based on bidirectional long short-term memory (BiLSTM) and are formulated for two categories: sequence labeling (Stanovsky et al., 2018; Sarhan and Spruit, 2019; Jia and Xiang, 2019) and se- quence generation (Cui et al., 2018; Sun et al., 2018; Bhutani et al., 2019). The latter enables flexi- ble extraction; however, it is more computationally expensive than the former. Additionally, generation methods …

Content truncated to 500 words