Robust Multi-bit Natural Language Watermarking through Invariant Features

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2092–2115 July 9-14, 2023 ©2023 Association for Computational Linguistics Robust Multi-bit Natural Language Watermarking through Invariant Features KiYoon Yoo1 Wonhyuk Ahn2 Jiho Jang1 Nojun Kwak1* 1Seoul National University 2Webtoon AI {961230,geographic,nojunk}@snu.ac.kr whahnize@gmail.com Abstract Recent years have witnessed a proliferation of valuable original natural language contents found in subscription-based media outlets, web novel platforms, and outputs of large language models. However, these contents are suscepti- ble to illegal piracy and potential misuse with- out proper security measures. This calls for a secure watermarking system to guarantee copyright protection through leakage tracing or ownership identification. To effectively com- bat piracy and protect copyrights, a multi-bit watermarking framework should be able to embed adequate bits of information and ex- tract the watermarks in a robust manner despite possible corruption. In this work, we explore ways to advance both payload and robustness by following a well-known proposition from image watermarking and identify features in natural language that are invariant to minor corruption. Through a systematic analysis of the possible sources of errors, we further pro- pose a corruption-resistant infill model. Our full method improves upon the previous work on robustness by +16.8% point on average on four datasets, three corruption types, and two corruption ratios.1 1 Introduction Recent years have witnessed a proliferation of orig- inal and valuable natural language contents such as those found in subscription-based media outlets (e.g. Financial Times, Medium), web novel plat- forms (e.g. Wattpad, Radish) – an industry that has shown rapid growth, especially in the East Asian market (HanSol, 2022; Zeyi, 2021) – and texts written by human-like language models (Ope- nAI, 2022; Chiang et al., 2023; Taori et al., 2023). Without proper security measures, however, these contents are susceptible to illegal piracy and dis- tribution, financially damaging the creators of the 1Department of Intelligence and Information, Graduate School of Convergence Science and Technology. https://github.com/bangawayoo/nlp-watermarking content and the market industry. In addition, the recent emergence of human-like language models like ChatGPT has raised concerns regarding the mass generation of disinformation (Goldstein et al., 2023). This calls for a secure watermarking system to guarantee copyright protection or detect misuse of language models. Digital watermarking is a technology that en- ables the embedding of information into multime- dia (e.g. image, video, audio) in an unnoticeable way without degrading the original utility of the content. Through embedding information such as owner/purchaser ID, its application includes leak- age tracing, ownership identification, meta-data binding, and tamper-proofing. To effectively com- bat intentional evasion by the adversary or uninten- tional digital degradation, a watermarking frame- work should not only be able to embed adequate bits of information but also demonstrate robustness against potential corruption (Tao et al., 2014; Zhu et al., 2018). Watermarking in image and video con- tents has been extensively explored for pre-deep learning methods (Hsu and Wu, 1999; Wolfgang et al., 1999; Wang et al., 2001). With the advent of deep neural networks, deep watermarking …

Content truncated to 500 words