MITRE at SemEval-2017 Task 1: Simple Semantic Similarity

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 185–190, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics MITRE at SemEval-2017 Task 1: Simple Semantic Similarity John Henderson, Elizabeth M. Merkhofer, Laura Strickhart and Guido Zarrella The MITRE Corporation 202 Burlington Road Bedford, MA 01730-1420, USA {jhndrsn,emerkhofer,lstrickhart,jzarrella}@mitre.org Abstract This paper describes MITRE’s participa- tion in the Semantic Textual Similarity task (SemEval-2017 Task 1), which eval- uated machine learning approaches to the identification of similar meaning among text snippets in English, Arabic, Spanish, and Turkish. We detail the techniques we explored, ranging from simple bag-of- ngrams classifiers to neural architectures with varied attention and alignment mech- anisms. Linear regression is used to tie the systems together into an ensemble submit- ted for evaluation. The resulting system is capable of matching human similarity rat- ings of image captions with correlations of 0.73 to 0.83 in monolingual settings and 0.68 to 0.78 in cross-lingual conditions. 1 Introduction Semantic Textual Similarity (STS) measures the degree to which two snippets of text convey the same meaning. Cross-lingual STS measures the same for sentence pairs written in two different languages. Automatic identification of semanti- cally similar text has practical applications in do- mains such as evaluation of machine translation outputs, discovery of parallel sentences in com- parable corpora, essay grading, and news summa- rization. It serves as an easily explained assay for systems modeling semantics. SemEval-2017 marked the sixth consecutive year of a shared task measuring progress in STS. Current machine learning approaches to measur- ing semantic similarity vary widely. One de- sign decision for STS systems is whether to ex- plicitly align words between paired sentences. Wieting et al. (2016) demonstrate that sentence embeddings without explicit alignment or atten- tion can often provide reasonable performance on STS tasks. Related work in textual entail- ment offers evidence that neural models with soft alignment outperform embeddings-only ap- proaches Chen et al. (2016); Parikh et al. (2016). However these results were obtained on a dataset multiple orders of magnitude larger than existing STS datasets. In absence of large datasets, word alignments similar to those used in statistical ma- chine translation have proven to be useful (Zarrella et al., 2015; Itoh, 2016). In this effort we explored diverse methods for aligning words in pairs of candidate sentences: translation-inspired hard word alignments as well as soft alignments learned by deep neural net- works with attention. We also examined a variety of approaches for comparing aligned words, rang- ing from bag-of-ngrams features leveraging hand- engineered lexical databases, to recurrent and con- volutional neural networks operating over dis- tributed representations. Although an ideal cross- lingual STS system might operate directly on input sentences in their original language, we used ma- chine translation to convert all the inputs into En- glish. The paucity of in-domain training data and the simplicity of the image caption genre made the translation approach reasonable. Our contri- bution builds on approaches developed for En- glish STS but points a way forward for …

Content truncated to 500 words