Are Emoji, Sentiment, and Emotion Friends? A Multi-task Learning for Emoji, Sentiment, and Emotion Analysis

Integrating Label Attention into CRF-based Vietnamese Constituency Parser Duy Vu-Tran, Phu-Thinh Pham, Duc Do, An-Vinh Luong and Dien Dinh University of Science, Ho Chi Minh city, Vietnam Vietnam National University, Ho Chi Minh city, Vietnam {vtduy18, phpthinh18}@apcs.fitus.edu.vn {dotrananhduc, anvinhluong}@gmail.com ddien@fit.hcmus.edu.vn Abstract Attention mechanisms and linear-chain con- ditional random field (CRF) have been ap- plied to constituency parsing, and the achieved results are phenomenal. While self-attention and label attention layers (LAL) have been proven to be state-of-the-arts in English con- stituency parsing for their improvement in the encoding phase, the CRF two-stage technique shows its effectiveness in lowering computa- tional cost. Attention-based architectures al- low a word (self-attention) or a label (label- attention) to include its own viewpoint into ex- tracted information. Our system is an extension of the current CRF-based model with additional attention-based methods to improve the quality of the encoding phase. Another crucial factor in our encoder is BERT as the pre-trained model has gained recognition in various natural lan- guage processing (NLP) tasks. Taking the ad- vantage of different methods, we implement a model that combines label attention, contextu- alized encoding, and conditional random field. Furthermore, we adopt the biaffine attention, which is mainly used in the dependency pars- ing task, in our scoring layer. The architecture performs greatly on the Vietnamese treebank as it gives an over-85 F1-score on the test set and an over-82 F1-score on the dev set. On a larger scale, our idea of integration could be utilized in other language models. 1 Introduction In the modern era, syntactical parsing has gained remarkable results due to the rise of deep learn- ing and neural networks (Mrini et al., 2020; Zhang et al., 2020; Zhou and Zhao, 2019; Stern et al., 2017; Wang and Tu, 2020; Yang and Deng, 2020). Especially, constant improvements on English and Chinese constituency parsing come from propos- als of machine learning techniques (Mrini et al., 2020; Zhang et al., 2020; Zhou and Zhao, 2019; Stern et al., 2017), which encourages us to apply such models to the same task of the Vietnamese language. Since most of the existing constituency parsers are encoder-decoder architectures, the main approach to improving the performance is to up- grade either encoder or decoder or both. Encoders handle inputs and extract their signifi- cance in vector forms so that the model can easily understand them. Particularly, the inputs of a con- stituency parser are sentences, and the encoders try to learn the information of each word or span of the sentences. Recurrent neural networks (RNN) and Long short-term memory networks (LSTM) are the main tools to extract such features from data for their ability to learn the contextual characteristics, and Zhang et al. (2020); Stern et al. (2017); Gaddy et al. (2018) have benefited from these mecha- nisms. PHAN et al. (2019), one of the pioneers in Vietnamese constituency parsing, used BiLSTM in their encoder. Despite RNNs and LSTMs’ ability to capture contextual information, they are surpassed by the works of self-attention. In 2017, Vaswani …

Content truncated to 500 words