A Large-Scale Chinese Multimodal NER Dataset with Speech Clues

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2807–2818 August 1–6, 2021. ©2021 Association for Computational Linguistics 2807 A Large-Scale Chinese Multimodal NER Dataset with Speech Clues Dianbo Sui, Zhengkun Tian, Yubo Chen, Kang Liu, Jun Zhao National Laboratory of Pattern Recognition, Institute of Automation, CAS School of Artificial Intelligence, University of Chinese Academy of Sciences {dianbo.sui, zhengkun.tian, yubo.chen, kliu, jzhao}@nlpr.ia.ac.cn Abstract In this paper, we aim to explore an uncharted territory, which is Chinese multimodal named entity recognition (NER) with both textual and acoustic contents. To achieve this, we con- struct a large-scale human-annotated Chinese multimodal NER dataset, named CNERTA. Our corpus totally contains 42,987 anno- tated sentences accompanying by 71 hours of speech data. Based on this dataset, we propose a family of strong and representative base- line models, which can leverage textual fea- tures or multimodal features. Upon these base- lines, to capture the natural monotonic align- ment between the textual modality and the acoustic modality, we further propose a sim- ple multimodal multitask model by introduc- ing a speech-to-text alignment auxiliary task. Through extensive experiments, we observe that: (1) Progressive performance boosts as we move from unimodal to multimodal, verifying the necessity of integrating speech clues into Chinese NER. (2) Our proposed model yields state-of-the-art (SoTA) results on CNERTA, demonstrating its effectiveness. For further re- search, the annotated dataset is publicly avail- able at http://github.com/DianboWork/ CNERTA. 1 Introduction “Speech is a part of thought.” — Oliver Sacks, Seeing Voices As a fundamental subtask of information extraction, named entity recognition (NER) aims to locate and classify named entities mentioned in unstructured texts into predefined semantic categories, such as person names, locations and organizations. NER plays a crucial role in many natural language pro- cessing (NLP) tasks, including relation extraction (Zelenko et al., 2003), question answering (Moll´a et al., 2006) and summarization (Aramaki et al., 2009). 南京市 南京市(LOC) Nanjing city 长江大桥(LOC) Yangtze River Bridge 南 京 市 长 江 大 桥 The Nanjing Yangtze River Bridge Speech: Sentence: 南京市 南京(LOC) Nanjing 江大桥(PER) Daqiao Jiang 南 京 市 长 江 大 桥 Nanjing Mayor Daqiao Jiang Speech: Sentence: Sentence: 南 京 市 长 江 大 桥 South Capital City Long Rive Big Bridge Figure 1: The given sentence “南京市长江大桥” can be segmented into “[南京市] [长江大桥]” or “[南京] [市长] [江大桥]”. Only based on textual contents, it is difficult to infer NER tags. But the speech waveforms of these two segmentations are radically different. Most of the research on NER, such as Lam- ple et al. (2016); Ma and Hovy (2016); Chiu and Nichols (2016), only relies on the textual modal- ity to infer tags. However, when texts are noisy or short, and it is not sufficient to locate and clas- sify named entities accurately only based on tex- tual information (Baldwin et al., 2015; Lu et al., 2018). One promising solution is to introduce other modalities as the supplement of the textual modal- ity. So …

Content truncated to 500 words