Insights from Gathering MT Productivity Metrics at Scale

Internal Use - Confidential Insights from Gathering MT Productivity Metrics at Scale Georg Kirchner Dell Technologies 42 South Street Hopkinton, MA, 01748, U.S.A. Georg.Kirchner@dell.com Abstract In this paper, we describe Dell EMC’s framework to automatically collect MT- related productivity metrics from a large translation supply chain over an extended period of time, the characteristics and volume of the gathered data, and the in- sights from analyzing the data to guide our MT strategy. Aligning tools, processes and people re- quired decisions, concessions and contri- butions from Dell management, technol- ogy providers, tool implementors, LSPs and linguists to harvest data at scale over 2+ years while Dell EMC migrated from customized SMT to generic NMT and then customized NMT systems. For content in two quality tiers, we ranked language pairs by productivity, graphed trendlines, compared the time needed to edit machine translations ver- sus fuzzy matches, studied the time spent on segments with no post-edits, and go- ing by the post-edit density, reviewed segment distribution on a post-edit scale of 1 to 10 and any correlation between the extent of edits and segment length. 1 Gathering Data at Scale Dell’s translation efforts produce significant amounts of linguistic data. Getting to the data, however, is not trivial since it originates with hundreds of linguists who are one or two organi- zational layers removed in Dell’s external supply chain. Each linguist may prefer a different CAT © 2020 The authors. This article is licensed under a Creative Commons 3.0 license, no derivative works, at- tribution, CC-BY-ND. tool, with or w/o the necessary features to track metrics for productivity or quality. Especially if desktop CAT tools require manual configuration from linguists, the constant churn in the resource pool makes it difficult to collect data reliably over time. For various operational needs, when looking for alternatives six years ago, we qualified trans- lation technology and implemented it as a col- laborative environment for linguists to share data in real time. After we had integrated this envi- ronment, GlobalLink1, with the TAUS DQF Dashboard2, we were ready to harvest metrics on productivity from our Microsoft Translator3 MT systems automatically between August 2017 and February 2020. 2 The Metrics We will discuss the following MT metrics: productivity and post-edits. We chose to measure both metrics at the first linguistic step, Translation, although our linguists may have made further changes downstream at the Editing, Proofing, Client Review and Feedback Implementation steps. The TAUS DQF Dashboard expresses produc- tivity in words post-edited per hour. This number is calculated from the number of words in seg- ments and the milliseconds these segments are active in the CAT tool for editing. As for post-edits (PED), the TAUS DQF Dashboard distinguishes between post-edit densi- ty (PEDe) and post-edit distance (PEDi). Both are calculated with the Levenshtein algorithm (1966). 1 https://www.translations.com/globallink/index.html 2 https://qd.taus.net/ 3 https://www.microsoft.com/en- us/translator/business/translator-api/ Internal Use - Confidential PEDe expresses changes across the entire sample in percentages, in the average number of characters changed per 100 characters. At the segment level, …

Content truncated to 500 words