Lisbon: Evaluating TurboSemanticParser on Multiple Languages and Out-of-Domain Data

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 970–973, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics Lisbon: Evaluating TurboSemanticParser on Multiple Languages and Out-of-Domain Data Mariana S. C. Almeida∗† Andr´e F. T. Martins∗† ∗Priberam Labs, Alameda D. Afonso Henriques, 41, 2o, 1000-123 Lisboa, Portugal †Instituto de Telecomunicac¸˜oes, Instituto Superior T´ecnico, 1049-001 Lisboa, Portugal {atm,mla}@priberam.pt Abstract As part of the SemEval-2015 shared task on Broad-Coverage Semantic Dependency Pars- ing, we evaluate the performace of our last year’s system (TurboSemanticParser) on mul- tiple languages and out-of-domain data. Our system is characterized by a feature-rich lin- ear model, that includes scores for first and second-order dependencies (arcs, siblings, grandparents and co-parents). For decoding this second-order model, we solve a linear re- laxation of that problem using alternating di- rections dual decomposition (AD3). The ex- periments have shown that, even though the parser’s performance in Chinese and Czech at- tains around 80% (not too far from English performance), domain shift is a serious issue, suggesting domain adaptation as an interest- ing avenue for future research. 1 Introduction The last years have witnessed a continuous progress in statistical multilingual models for syntax, thanks to shared tasks such as CoNLL 2006-7 (Buchholz and Marsi, 2006; Nivre et al., 2007) and, more re- cently, SPMRL 2013-14 (Seddah et al., 2013; Sed- dah et al., 2014). As a global trend, we observe that models that incorporate rich global features are typically more accurate, even if pruning is neces- sary or decoding needs to be approximate (McDon- ald et al., 2006; Koo and Collins, 2010; Bohnet and Nivre, 2012; Martins et al., 2009, 2013). The same rationale applies to semantic dependency parsing, also a structured prediction problem, but where the output variable is a semantic graph, rather than a syntactic tree. Indeed, the best performing systems in last year shared task on broad-coverage seman- tic dependency parsing follow this principle (Oepen et al., 2014). This year, a new challenge was put forth: how to handle multiple languages and out-of- domain data? Our proposed parser (§2) is essentially the same that we submitted in the previous year to the same SemEval task (Martins and Almeida, 2014), where we scored top in the open challenge and second in the closed track. This year, we report results using new out-of-domain and multilingual data (namely, Czech and Chinese, in addition to English). For the English language, we participated in the closed and open tracks, using as additional resources the syn- tactic dependency annotations provided by the orga- nizers. For Czech and Chinese, we only addressed the closed track, since no companion data were pro- vided for these languages. We did not participate in the gold track that uses gold-standard syntactic an- notations; and we did not address the prediction of predicate senses. 2 Semantic Parser For this year’s shared task, we re-run the semantic parser that we developed last year, which is fully desc1ribed in Martins and Almeida (2014), on the new datasets. Since this parser …

Content truncated to 500 words