Fine-grained Information Extraction from Biomedical Literature based on Knowledge-enriched Abstract Meaning Representation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 577–589 August 1–6, 2021. ©2021 Association for Computational Linguistics 577 COSY: COunterfactual SYntax for Cross-Lingual Understanding Sicheng Yu1, Hao Zhang2,3, Yulei Niu2, Qianru Sun1, Jing Jiang1 1Singapore Management University, Singapore 2Nanyang Technological University, Singapore 3Agency for Science, Technology and Research, Singapore scyu.2018@phdcs.smu.edu.sg, hao007@e.ntu.edu.sg yn.yuleiniu@gmail.com, {qianrusun,jingjiang}@smu.edu.sg Abstract Pre-trained multilingual language models, e.g., multilingual-BERT, are widely used in cross- lingual tasks, yielding the state-of-the-art per- formance. However, such models suffer from a large performance gap between source and target languages, especially in the zero-shot setting, where the models are fine-tuned only on English but tested on other languages for the same task. We tackle this issue by incorpo- rating language-agnostic information, specifi- cally, universal syntax such as dependency re- lations and POS tags, into language models, based on the observation that universal syn- tax is transferable across different languages. Our approach, named COunterfactual SYn- tax (COSY), includes the design of SYntax- aware networks as well as a COunterfactual training method to implicitly force the net- works to learn not only the semantics but also the syntax. To evaluate COSY, we con- duct cross-lingual experiments on natural lan- guage inference and question answering using mBERT and XLM-R as network backbones. Our results show that COSY achieves the state- of-the-art performance for both tasks, without using auxiliary dataset.1 1 Introduction With the emergence of BERT (Devlin et al., 2019), large-scale pre-trained language models have become an indispensable component in the solutions to many natural language process- ing (NLP) tasks. Recently, large-scale multilingual transformer-based models, such as mBERT (Devlin et al., 2019), XLM (Lample and Conneau, 2019) and XLM-R (Conneau et al., 2020a), have been widely deployed as backbones in cross-lingual NLP tasks (Wu and Dredze, 2019; Pires et al., 2019; Ke- ung et al., 2019). However, these models trained 1Our code is publicly available on GitHub: https:// github.com/PluviophileYU/COSY English: I bought two new laptops yesterday . bought �� I � laptops �� yesterday �� . � new �� two �� <ROOT> <OBJ> Chinese: � �� �� �� �� �� � [PRON] [VERB] [NOUN] [NOUN] [PUNCT] [NUM] [ADJ] Shared Syntax: Figure 1: Examples of two sentences in English and Chinese that have the same meaning and share the same syntax in the format of dependency relations and POS tags. on a single resource-rich language, e.g., English, all suffer from a large drop of performance when tested on different target languages, e.g., Chinese and German—where the setting is called zero- shot cross-lingual transfer. For example, on the XQUAD dataset, mBERT achieves a 24 percent- age points lower exact match score on the target language Chinese than on the training language English (Hu et al., 2020). This indicates that this model has seriously overfitted English. An intuitive way to tackle this is to introduce language-agnostic information—the most transfer- able feature across languages, which is lacking in existing multilingual language models (Choenni and Shutova, 2020). …

Content truncated to 500 words