Towards relation extraction from speech

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10751 - 10762 December 7-11, 2022 ©2022 Association for Computational Linguistics Towards Relation Extraction from Speech Tongtong Wu∗,♢,△ Guitao Wang∗,♢ Jinming Zhao∗,△ Zhaoran Liu† Guilin Qi♢ Yuan-Fang Li△ Gholamreza Haffari△ ♢Southeast University, China; †Zhejiang University, China; △Monash University, Australia ♢{wutong8023,220222117,gqi}@seu.edu.cn, †jiuyinlau@gmail.com △{first_name.last_name}@monash.edu Abstract Relation extraction has focused on extracting semantic relationships between entities from the unstructured written textual data. However, with the vast and rapidly increasing amounts of spoken data, relation extraction from speech is an important but under-explored problem. In this paper, we propose a new information extrac- tion task, speech relation extraction (SpeechRE). To facilitate further research, we construct the first synthetic training datasets, as well as the first human-spoken test set with native English speakers. We establish strong baseline perfor- mance for SpeechRE via two approaches. The pipeline approach connects a pretrained ASR module with a text-based relation extraction module. The end-to-end approach employs a cross-modal encoder-decoder architecture. Our comprehensive experiments reveal the relative strengths and weaknesses of these approaches, and shed light on important future directions in SpeechRE research. We share the source code and datasets on https://github.com/ wutong8023/SpeechRE. 1 Introduction Relation extraction (RE) (Han et al., 2020) is an important information extraction task, which aims at extracting structured semantic relations between entities from unstructured data, typically text. Besides text, there is also a plethora of speech data that is being continually produced. These include news reports, interviews, meetings and dialogues, to name a few. Extracting relations from speech is an important but under-explored problem. In this work, we take the first step towards addressing relation extraction (RE) from speech, introducing a new information extraction task, Speech Relation Extraction (SpeechRE). The input for this task is raw audio and the output is one or more triplets, each of which representing a relation ∗ denotes the equal contribution. Figure1: Thecomparisonamongconventionaltranscript- based relation extraction, ASR outputs-based relation extraction, and the end-to-end speech relation extraction. between a pair of two entities appearing in the speech, e.g., [entity1, relation, entity2]. SpeechRE and text-based RE (TextRE) both involve content understanding. The former is more challenging than the latter, mainly due to the char- acteristics of speech. (i) Speech carries much richer information beyond linguistic content (unlike text), for instance, emotion, speaker style and background noise; and it is non-trivial to disentangle the content element (Mohamed et al., 2022), which is needed for SpeechRE. (ii) Speech is continuous without sequence/word boundaries, implying the difficulty of determining the exact audio spans for target words (entity and relation). (iii) Audio signals are orders of magnitude longer than the corresponding transcripts, which makes speech encoding for long-span extraction more challenging due to more demanding hardware requirements, especially with Transformer (Vaswani et al., 2017). In the absence of SpeechRE training data, we construct three benchmark datasets for this task by converting two commonly used TextRE 10751 datasets (i.e., CoNLL04 (Roth and Yih, 2004a) and ReTACRED (Stoica et al., 2021a) to speech with a …

Content truncated to 500 words