Improving a Neural-based Tagger for Multiword Expressions Identification

Improving a Neural-based Tagger for Multiword Expression Identification Duˇsan Variˇs, Natalia Klyueva Institute of Formal and Applied Linguistics, Charles University; The Hong Kong Polytechnic University Malostransk´e n´amˇest´ı 25, Prague; 11 Yuk Choi Rd, Hung Hom, varis@ufal.mff.cuni.cz, natalia.klyueva@polyu.edu.hk Abstract In this paper, we present a set of improvements introduced to MUMULS, a tagger for the automatic detection of verbal multiword expressions. Our tagger participated in the PARSEME shared task and it was the only one based on neural networks. We show that character-level embeddings can improve the performance, mainly by reducing the out-of-vocabulary rate. Furthermore, replacing the softmax layer in the decoder by a conditional random field classifier brings additional improvements. Finally, we compare different context-aware feature representations of input tokens using various encoder architectures. The experiments on Czech show that the combination of character-level embeddings using a convolutional network, self-attentive encoding layer over the word representations and an output conditional random field classifier yields the best empirical results. Keywords: multiword expressions, machine learning, deep learning, conditional random field 1. Introduction Multiword Expression (MWE) is a sequence of words for which the meaning of a whole sequence cannot be derived from the meaning of its components straightforwardly. MWEs are viewed by computational linguists as a “pain in the neck of NLP” due to their non-compositionality and irregularity that can cause problems in areas such as ma- chine translation, terminology extraction, etc. Regarding this, MWEs are largely addressed in both the theoretical and applied research. Associative measures (calculating association between distinct words in MWE) are usually used for extracting MWEs (Ramisch, 2015; Kilgarriff et al., 2014). The identification of MWE in the text is thus a challenging task. Within PARSEME’s special MWE-related project1, re- searchers from different countries created guidelines on how to define MWEs in the text and annotated corpora in 18 languages. The focus was on verbal MWEs which were categorized into five classes: idioms (ID), light verb con- structions (LVC), inherently reflexive verbs (IReflV), verb- particle constructions (VPC), and other (OTH). The process of annotation was language-dependent. For instance, in Hungarian, VPCs are annotated and IReflVs are not, how- ever, in Czech it is the other way around. These data then served as the training data for systems par- ticipating in the shared task on automatic verbal multiword expression (VMWE) identification (Savary et al., 2017). In addition to the MWEs markings, morphosyntactic mark- ings were provided in the corpora as well. Seven systems based on various approaches and algorithms participated in the task. Two of them were based on condi- tional random field (CRF) (Maldonado et al., 2017; Boros¸ et al., 2017), the other was trained using dependency pars- ing (Simk´o et al., 2017) and the winner was a transition- based system exploiting syntactic rules (Al Saied et al., 2017). For some of the languages, our previous model based on the neural networks (NN) had comparable scores with other 1https://typo.uni-konstanz.de/parseme/ multilingual systems, yet it performed best only in one lan- guage – Romanian. Later comparison between our system and …

Content truncated to 500 words