Large-Scale Hierarchical Alignment for Data-driven Text Rewriting

Proceedings of Recent Advances in Natural Language Processing, pages 844–853, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_098 844 Large-Scale Hierarchical Alignment for Data-driven Text Rewriting Nikola I. Nikolov and Richard H.R. Hahnloser Institute of Neuroinformatics, University of Z¨urich and ETH Z¨urich, Switzerland {niniko, rich}@ini.ethz.ch Abstract We propose a simple unsupervised method for extracting pseudo-parallel monolin- gual sentence pairs from comparable cor- pora representative of two different text styles, such as news articles and scien- tific papers. Our approach does not re- quire a seed parallel corpus, but instead relies solely on hierarchical search over pre-trained embeddings of documents and sentences. We demonstrate the effective- ness of our method through automatic and extrinsic evaluation on text simplification from the normal to the Simple Wikipedia. We show that pseudo-parallel sentences extracted with our method not only sup- plement existing parallel data, but can even lead to competitive performance on their own.1 1 Introduction Parallel corpora are indispensable resources for advancing monolingual and multilingual text rewriting tasks. Due to the scarce availability of parallel corpora, and the cost of manual creation, a number of methods have been proposed that can perform large-scale sentence alignment: auto- matic extraction of pseudo-parallel sentence pairs from raw, comparable2 corpora. While pseudo- parallel data is beneficial for machine translation (Munteanu and Marcu, 2005), there has been little work on large-scale sentence alignment for mono- lingual text-to-text rewriting tasks, such as simpli- fication (Nisioi et al., 2017) or style transfer (Liu et al., 2016). Furthermore, the majority of existing methods (e.g. Marie and Fujita (2017); Gr´egoire 1Code available at https://github.com/ ninikolov/lha. 2Corpora that contain documents on similar topics. Source document Source document Target document Document alignment Sentence alignment Target 1 Target 2 Target 3 Target 4 Target 5 Figure 1: Illustration of large-scale hierarchical align- ment (LHA). For each document in a source dataset, doc- ument alignment retrieves matching documents from a tar- get dataset. In turn, sentence alignment retrieves matching sentence pairs from within each document pair. and Langlais (2018)) assume access to some par- allel training data. This impedes their application to cases where there is no parallel data available whatsoever, which is the case for the majority of text rewriting tasks, such as style transfer. In this paper, we propose a simple unsuper- vised method, Large-scale Hierarchical Align- ment (LHA) (Figure 1; Section 3), for extract- ing pseudo-parallel sentence pairs from two raw monolingual corpora which contain documents in two different author styles, such as scientific papers and press releases. LHA hierarchically searches for document and sentence nearest neigh- bors within the two corpora, extracting sentence pairs that have high semantic similarity, yet pre- serve the stylistic characteristics representative of their original datasets. LHA is robust to noise, fast and memory efficient, enabling its application to datasets on the order of hundreds of millions of sentences. Its generality makes it relevant to a wide range of monolingual text rewriting tasks. We demonstrate the effectiveness of LHA on automatic benchmarks for alignment (Section 4), as well as extrinsically, …

Content truncated to 500 words