Translation Memory Retrieval Using Lucene

Proceedings of Recent Advances in Natural Language Processing, pages 684–691 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_078 684 Abstract Translation Memory (TM) system, a major component of computer-assisted translation (CAT), is widely used to improve human translators’ productivity by making effective use of previously translated resource. We propose a method to achieve high-speed retrieval from a large translation memory by means of similarity evaluation based on vector model, and present the experimental result. Through our experiment using Lucene, an open source information retrieval search engine, we conclude that it is possible to achieve real-time retrieval speed of about tens of microseconds even for a large translation memory with 5 million segment pairs. 1 Introduction Translation memory technique is a key functionality being widely used in the field of CAT. Translation Memories (TMs) are “structured archives of past translations” which store pairs of corresponding text segments in source and target languages known as “translation units” (Simard, 2020). The size of translation memories and the quality of their contents are major impact factors crucial to the effectivity of the translation memory system which uses them. Due to the importance of translation memories, there has been done lots of research work for building large TMs on world- wide scale, not just in individual countries (Steinberger et al. 2012). What plays an important role for TM system is also the similarity evaluation between the input sentence to be translated and the source segment in the TM. The main task of TM system is to get a translation unit whose source segment is the most similar to the input sentence out of TM. There are two possible solutions in performing the task: One solution is to adopt an intelligent TM matching mechanism which is able to correctly calculate the similarity between the input sentence and the source segment in the translation memory. The other solution is to increase the size of TM by collecting translated resources as much as possible. No matter how intelligent the TM matching mechanism is, small-size TM cannot afford rich performance. Of course, the choice of TM matching method is important for improving the effectivity of TM system. But what is no less important than any TM matching method is to use a reasonable size TM. The larger the TM, the higher the possibility to get a translation unit whose source segment is very similar to the input sentence out of the TM. In general, the main value of a TM consists in the number of segments - its size. However, large TMs automatically lead to slow response times. A slow TM might actually slow down a translator, so that fast response time is an essential characteristic of any TM. Many research works have been reported to improve TM matching and retrieval, but the majority of those approaches were just evaluated on relatively small TMs. To our best knowledge, the largest TM tested so far in previous research works is the first five parts of the 2013 DGT-TM (which consisted of 305,324 segment pairs) …

Content truncated to 500 words