Hierarchical Document Encoder for Parallel Corpus Mining

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 64–72 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 64 Hierarchical Document Encoder for Parallel Corpus Mining Mandy Guo, Yinfei Yang, Keith Stevens, Daniel Cer, Heming Ge, Yun-Hsuan Sung, Brian Strope, Ray Kurzweil Google AI 1600 Amphitheatre Parkway Mountain View, CA, USA {xyguo, yinfeiy, kstevens, cer, hemingge, yhsung, raykurzweil}@google.com Abstract We explore using multilingual document em- beddings for nearest neighbor mining of par- allel data. Three document-level representa- tions are investigated: (i) document embed- dings generated by simply averaging multilin- gual sentence embeddings; (ii) a neural bag- of-words (BoW) document encoding model; (iii) a hierarchical multilingual document en- coder (HiDE) that builds on our sentence-level model. The results show document embed- dings derived from sentence-level averaging are surprisingly effective for clean datasets, but suggest models trained hierarchically at the document-level are more effective on noisy data. Analysis experiments demonstrate our hierarchical models are very robust to vari- ations in the underlying sentence embedding quality. Using document embeddings trained with HiDE achieves state-of-the-art perfor- mance on United Nations (UN) parallel doc- ument mining, 94.9% P@11 for en-fr and 97.3% P@1 for en-es. 1 Introduction Obtaining a high-quality parallel training corpus is one of the most critical issues in machine trans- lation. Previous work on parallel document min- ing using large distributed systems has proven effective (Uszkoreit et al., 2010; Antonova and Misyurev, 2011), but these systems are often heav- ily engineered and computationally intensive. Re- cent work on parallel data mining has focused on sentence-level embeddings (Guo et al., 2018; Artetxe and Schwenk, 2018; Yang et al., 2019). However, these sentence embedding methods have had limited success when applied to document- level mining tasks (Guo et al., 2018). A re- cent study from Yang et al. (2019) shows that 1We use evaluation metrics precision at N, here P@1 means precision at 1 document embeddings obtained from averaging sentence embeddings can achieve state-of-the-art performance in document retrieval on the United Nation (UN) corpus. This simple averaging ap- proach, however, heavily relies on high quality sentence embeddings and the cleanliness of doc- uments in the application domain. In our work, we explore using three variants of document-level embeddings for parallel document mining: (i) simple averaging of embeddings from a multilingual sentence embedding model (Yang et al., 2019); (ii) trained document-level embed- dings based on document unigrams; (iii) a simple hierarchical document encoder (HiDE) trained on documents pairs using the output of our sentence- level model. The results show document embeddings are able to achieve strong performance on parallel document mining. On a test set mined from the web, all models achieve strong retrieval perfor- mance, the best being 91.4% P@1 for en-fr and 81.8% for en-es from the hierarchical document models. On the United Nations (UN) document mining task (Ziemski et al., 2016), our best model achieves 96.7% P@1 for en-fr and 97.3% P@1 for en-es, a 3%+ absolute improvement over the prior state-of-the-art (Guo et al., …

Content truncated to 500 words