Open Aspect Target Sentiment Classification with Natural Language Prompts

Artificial Text Detection via Examining the Topology of Attention Maps Laida Kushnareva1∗, Daniil Cherniavskii2∗, Vladislav Mikhailov3∗, Ekaterina Artemova1,4, Serguei Barannikov2,5, Alexander Bernstein2, Irina Piontkovskaya1, Dmitri Piontkovski4, Evgeny Burnaev2 1Huawei Noah’s Ark lab, 2Skolkovo Institute of Science and Technology, 3SberDevices, 4HSE University, 5CNRS, IMJ Abstract The impressive capabilities of recent genera- tive models to create texts that are challeng- ing to distinguish from the human-written ones can be misused for generating fake news, prod- uct reviews, and even abusive content. Despite the prominent performance of existing meth- ods for artificial text detection, they still lack interpretability and robustness towards unseen models. To this end, we propose three novel types of interpretable topological features for this task based on Topological Data Analysis (TDA) which is currently understudied in the field of NLP. We empirically show that the features derived from the BERT model outper- form count- and neural-based baselines up to 10% on three common datasets, and tend to be the most robust towards unseen GPT-style gen- eration models as opposed to existing meth- ods. The probing analysis of the features re- veals their sensitivity to the surface and syn- tactic properties. The results demonstrate that TDA is a promising line with respect to NLP tasks, specifically the ones that incorporate sur- face and structural information. 1 Introduction Recent text generation models (TGMs) based on the transformer architecture (Vaswani et al., 2017) have demonstrated impressive capabilities of cre- ating texts which are very close to human in terms of fluency, coherence, grammatical and factual correctness (Keskar et al., 2019; Zellers et al., 2020; Yang et al., 2019). Extensive GPT-style TGMs (Radford et al., 2018) have achieved out- standing results over a great scope of NLP tasks employing zero-shot, one-shot, and few-shot tech- niques, even outperforming state-of-the-art fine- tuning approaches (Brown et al., 2020). However, such models can be misused for generating fake news (Zellers et al., 2020; Uchendu et al., 2020), product reviews (Adelani et al., 2020), and even ∗Equal contribution. extremist and abusive content (McGuffie and New- house, 2020). Many attempts have been made to develop artificial text detectors (Jawahar et al., 2020), ranging from classical ML methods over count- based features (Uchendu et al., 2019) to advanced transformer-based models (Adelani et al., 2020) and unsupervised approaches (Solaiman et al., 2019). Despite the prominent performance of these methods across various domains, they still lack in- terpretability and robustness towards unseen mod- els. This paper introduces a novel method for artifi- cial text detection based on Topological Data Anal- ysis (TDA) which has been understudied in the field of NLP. The motivation behind this approach relies on the fact that (i) the attention maps gener- ated by the transformer model can be represented as weighted bipartite graphs and thus can be effi- ciently investigated with TDA, (ii) TDA methods are known to capture well surface and structural patterns in data which, we believe, are crucial to the task. The contributions are summarized as follows. (i) To the best of our knowledge, this …

Content truncated to 500 words