HMIST: Hierarchical Multilingual Isometric Speech Translation using Multi-Task Learning Framework and it’s influence on Automatic Dubbing

HMIST: Hierarchical Multilingual Isometric Speech Translation using Multi-Task Learning Framework for Automatic Dubbing Nidhir Bhavsar$, Aakash Bhatnagar&, Muskaan Singh# and Petr Motlicek# $University of Potsdam, Potsdam, Germany & Boston University, Boston, Massachusetts # Speech and Audio Processing Group, IDIAP Research Institute, Martigny, Switzerland bhavsar@uni-potsdam.de, aakash07@bu.edu, (msingh,petr.motlicek)@idiap.ch Abstract In this paper, we present an approach and im- pact of isometric neural machine translation on the automatic dubbing process. The length of generated isometric translated sentences ranges within a ±10% of the source text. We propose a hierarchical and multilingual approach toward generating isometric translation via publicly available MUST-C, WMT, and IIT-Bombay(en- hi) datasets. Our experiments use namely, Ger- man(de), French(fr), Russian(ru), Italian(it), and Hindi(hi) languages. Additionally, we im- plement a paraphrasing module with Opuspar- cus(fr,de,ru), PAWS-X(fr,de) and Topaco(ru) datasets for German, French, and Russian lan- guages to enhance the vocabulary and main- tain the isometric constraints. In performance analysis, we report the average length range of source to translation, 55.15% for all languages, while ru exhibits the highest with 62.475% and a relative improvement of 23.04% from the baseline OPUS-MT model. 1 Introduction Isometric translation is a relatively new concept in neural machine translation. As video content reaches worldwide, it becomes crucial to localize it for different regions. One of the major problems faced while translating media content is the syn- chrony between the translated output and the visual content. This problem mainly occurs due to a con- siderable variation in vocabulary between different languages. [1] state that the ideal length of the generated output should be within ±10% range of the source length. The recent machine translation models do not have any parameters to control the length of the output sequences. To solve the problem mentioned earlier, we fine- tune different pre-train language models using the prompt engineering method. The initial step in all our methods is to identify the appropriate prompt. We use the approach described in [2] to generate prompts while training. Prompt engineering is an efficient way to perform transfer learning while fine-tuning a model. In this paper, we present, an empirical analy- sis of our different translation and paraphrasing models. In our approach, the best performing trans- lation model is OPUS MT and the most efficient paraphrasing model is mBART [3]. However, we train the paraphrasing model only for German, Rus- sian and French because of the limited number of languages supported by paraphrasing datasets. We use a combination of Opusparcus and Topaco datasets for Russian, and Opusparcus and PAWS-X datasets for French and German. As per our knowl- edge, while writing this paper, we cannot find a standard paraphrasing corpus for Italian and Hindi languages. 2 Background In this section, we further explain neural machine translation in section 2.1, and its corresponding controlling output length in section 2.2 with lexical and length constraint. 2.1 Neural Machine Translation For a language pair with parallel data as 1, an MT model parameterized with θ, trains to maximize likelihood on the training sample pairs 2. D = {(si, …

Content truncated to 500 words