Enhancing Language Representation with Constructional Information for Natural Language Understanding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4685–4705 July 9-14, 2023 ©2023 Association for Computational Linguistics Enhancing Language Representation with Constructional Information for Natural Language Understanding Lvxiaowei Xu, Jianwang Wu, Jiawei Peng, Zhilin Gong, Ming Cai∗, Tianxiang Wang Department of Computer Science and Technology, Zhejiang University {xlxw, wujw, pengjw, zhilingong, cm, wang_tx}@zju.edu.cn Abstract Natural language understanding (NLU) is an essential branch of natural language process- ing, which relies on representations generated by pre-trained language models (PLMs). How- ever, PLMs primarily focus on acquiring lexico- semantic information, while they may be un- able to adequately handle the meaning of con- structions. To address this issue, we introduce construction grammar (CxG), which highlights the pairings of form and meaning, to enrich lan- guage representation. We adopt usage-based construction grammar as the basis of our work, which is highly compatible with statistical mod- els such as PLMs. Then a HyCxG framework is proposed to enhance language representa- tion through a three-stage solution. First, all constructions are extracted from sentences via a slot-constraints approach. As constructions can overlap with each other, bringing redun- dancy and imbalance, we formulate the condi- tional max coverage problem for selecting the discriminative constructions. Finally, we pro- pose a relational hypergraph attention network to acquire representation from constructional information by capturing high-order word in- teractions among constructions. Extensive ex- periments demonstrate the superiority of the proposed model on a variety of NLU tasks. 1 Introduction Recent progress in natural language processing re- lies on pre-trained language models (PLMs) (De- vlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020; Raffel et al., 2020; He et al., 2021), which generate word representations by cap- turing syntactic and semantic features from their context. Typically, PLMs (e.g., BERT) employ the Masked Language Model (MLM) as a pre-training objective, randomly masking some tokens from the input before predicting the original tokens. How- ever, PLMs primarily focus on acquiring lexico- semantic information (Tayyar Madabushi et al., ∗ Corresponding author. Sentence Construction The [staff]N should be a bit more friendly. NOUN–AUX–be (staff–should–be) The restaurants try too hard to make fancy [food]N . ADV–hard–to (too–hard–to) I can understand the prices if it served better [food]N . if–PRON–VERB (if–it–served) Table 1: Visualization of attention scores from misclas- sified examples in aspect-based sentiment analysis. [·]N refers to the aspects with negative sentiment polarity. 2020) while the meaning is not attached to the words instantiating the construction, but rather to the abstract pattern itself (Weissweiler et al., 2023). Learning the representation of more complex and abstract linguistic units (called constructions) can be more challenging, such as collocations, argu- ment structures and sentence patterns. For exam- ple, the ditransitive construction (Goldberg, 1995), which involves the form “Subject–Verb–Object1– Object2”, denotes the meaning of transfer or giving of an entity (Object2) from an agent (Subject) to a recipient (Object1), such as “John gave Mary a book”. Linguistic experiments have proved that constructions can substantially contribute to …

Content truncated to 500 words