Bilingual Multi-word Expressions, Multiple-correspondence, and their cultivation from parallel patents: The Chinese-English case

Pay Attention to Categories: Syntax-Based Sentence Modeling with Metadata Projection Matrix Won Ik Cho and Nam Soo Kim Department of Electrical and Computer Engineering and INMC, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, Korea, 08826 wicho@hi.snu.ac.kr, nkim@snu.ac.kr Abstract Sentence modeling is a vital feature engineer- ing for document classification. Various fea- ture extraction and summarization algorithms have been adopted for efficient classification of a sentence, e.g., dense word vectors and neural network classifiers. Recently, the con- cept of attention for machine translation has been applied to various natural language pro- cessing (NLP) tasks and has shown signifi- cant performance. In this paper, we take a look at the syntactic categories of the words, to make up a metadata projection matrix that assigns strong restrictions on determining the attention weight. Unlike conventional atten- tion models, which are considered as a divi- sion of location-based approaches, our model adds a selection layer to highlight categori- cal metadata that may appear more than once. The proposed algorithm shows improved per- formance compared to the baselines with the tasks in syntax-semantics, suggesting a possi- bility of extension to other fields such as sym- bolic music or bitstream analysis. 1 Introduction Sentence modeling, which incorporates featuriza- tion and embedding, has been widely studied from short utterances to large-scale documents. Its use- fulness and broad applicability have been proven with various classification and regression tasks. Also, in recent years, attention models have demon- strated the significant performance of such ap- proaches, along with deep learning techniques that have shifted the paradigm of the standard recipes. In applying the attention models, we noted that the utility of the syntactic properties should be ex- plored in a bit wide point of view. Like the notes in music that have corresponding chords, the ob- servable components of a sentence are assigned syn- tactic categories after constituency parsing, such as noun, verb, and adjective. They are interpreted as a kind of metadata regarding each token1, that may appear more than once in the document. We want to claim such information can be exploited in mak- ing up the attention weight, not just being adopted as input-level data. For instance, in an oxymoron identification task (Cho et al., 2017), given a sen- tence like “This is a sugar-free sweet tea.”, it may be beneficial for the analysis to attend to sugar-free and sweet with a similar concentration, mainly due to their syntactic property being close to each other. Although such syntactic properties can be repre- sented in various ways such as tree structure and de- pendency, we pay attention to part-of-speech (POS), for some practicality. First of all, we already have many computationally efficient tools that can extract syntactic classes from the tokens of the sentence. Next, even though the POS tagger is not entirely ac- curate, the general tendency may provide sufficient information for classification. This flexibility can be supportive for the proposed model to analyze corpus with non-formal sentences such as tweets. The proposed model differs from the usual …

Content truncated to 500 words