Analyzing Challenges in Neural Machine Translation for Software Localization

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 211–231 May 2-6, 2023 ©2023 Association for Computational Linguistics FastKASSIM: A Fast Tree Kernel-Based Syntactic Similarity Metric Maximillian Chen∗, Caitlyn Chen,∗ Xiao Yu,∗ Zhou Yu Department of Computer Science, Columbia University, New York, NY maxchen@cs.columbia.edu {caitlyn.chen, xy2437, zy2461}@columbia.edu Abstract Syntax is a fundamental component of lan- guage, yet few metrics have been employed to capture syntactic similarity or coherence at the utterance- and document-level. The existing standard document-level syntactic similarity metric is computationally expensive and per- forms inconsistently when faced with syntacti- cally dissimilar documents. To address these challenges, we present FastKASSIM, a met- ric for utterance- and document-level syntactic similarity which pairs and averages the most similar constituency parse trees between a pair of documents based on tree kernels. FastKAS- SIM is more robust to syntactic dissimilarities and runs up to to 5.32 times faster than its prede- cessor over documents in the r/ChangeMyView corpus. FastKASSIM’s improvements allow us to examine hypotheses in two settings with large documents. We find that syntactically similar arguments on r/ChangeMyView tend to be more persuasive, and that syntax is predic- tive of authorship attribution in the Australian High Court Judgment corpus. 1 Introduction Syntax, the form of language, plays a crucial role in all aspects of natural language and communi- cation, whether explicitly or implicitly. In story- telling, writers often have their own styles rooted in different syntactic tendencies (Feng et al., 2012), allowing syntax to become indicators in prediction tasks such as gender (Sarawgi et al., 2011) and au- thorship (Raghavan et al., 2010) attribution. Syntax also has social connotations in different cultures — for example, in Russia, different social and de- mographic groups tend to use different syntactic patterns (Bogdanova-Beglarian et al., 2016). Such examples makes syntactic consistency crucial to capture in tasks such as machine translation and dialogue generation so that social conventions are not lost. Yet, recent research focuses primarily ∗denotes equal contribution. Utterance 1: When we hate, we always move away from the grace of God. When we become resentful and unforgiving, the world around us seems spiteful and meaningless. Utterance 2: How can you be skiing if you are already swimming? FastKASSIM Score: 0.219 CASSIM Score: 0.838 LSM Score: 0.623 Utterance 1: I like swimming because it is cool. Utterance 2: I love running because it is fun. FastKASSIM Score: 0.928 CASSIM Score: 0.962 LSM Score: 1.0 Figure 1: Comparison of FastKASSIM, CASSIM, and Linguistic Style Matching similarity scores. Top: two dissimilar utterances. Bottom: two similar utterances. All three metrics have strong agreement in cases of sim- ilar syntactic structure, but only FastKASSIM is able to recognize syntactically dissimilar utterances. The parse trees of these examples are visualized in Appendix D. on evaluating similarity and coherence in terms of dimensions like semantics, the meaning behind language, (with approaches such as BERT embed- dings (Reimers and Gurevych, 2019; Zhang et al., 2019b)), or lexical overlap (e.g., BLEU (Papineni et al., 2002)) — …

Content truncated to 500 words