LZ1904 at SemEval-2021 Task 5: Bi-LSTM-CRF for Toxic Span Detection using Pretrained Word Embedding

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1009–1014 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1009 LZ1904 at SemEval-2021 Task 5: Bi-LSTM-CRF for Toxic Span Detection using Pretrained Word Embedding Liang Zou Department of Mathematics New York University lz1904@nyu.edu Wen Li Department of Linguistics Indiana University wl9@indiana.edu Abstract Recurrent Neural Networks (RNN) have been widely used in various Natural Language Pro- cessing (NLP) tasks such as text classification, sequence tagging, and machine translation. Long Short Term Memory (LSTM), a special unit of RNN, has the advantage of memoriz- ing past and even future information in a sen- tence (especially for bidirectional LSTM). In the shared task of detecting toxic spans in texts, we first apply pretrained word embed- ding (GloVe) to generate the word vectors after tokenization. Then we construct Bidirectional Long Short Term Memory-Conditional Ran- dom Field (Bi-LSTM-CRF) model by Baidu research to predict whether each word in the sentence is toxic or not. We tune hyperparam- eters of dropout rate, number of LSTM units, embedding size with 10 epochs and choose the epoch with best validation recall. Our model achieves an F1 score of 66.99% on test dataset. 1 Introduction Detecting toxic words plays a critical role in so- cial media to ensure healthy online discussions. In previous study, some tasks (Liu et al., 2019; Borkan et al., 2019a) only identify offensive lan- guage based on the whole sentence or post. Most of them do not detect specific spans of words that make the sentence or post offensive. In SemEval-2021 Task 5: Toxic Spans Detection (Pavlopoulos et al., 2021), the data was collected from civil comments (Borkan et al., 2019b). Each post is in string format, and a word is marked as toxic span in the form of its characters’ offsets in the string. The goal of the task is to classify whether each word in a sentence is toxic or not. If so, the indices of characters in the word should be returned. The task is evaluated by F1 score based on the character offsets among all posts. The challenges of this task include: • The small dataset makes it very difficult to train complicated models like deep neural net- works, since it may cause overfitting. • We need to predict which word or phrase is toxic given a text (many-to-many) rather than whether the entire sentence is offensive or not (many-to-one). This creates restrictions on feature engineering and modeling: – Feature Engineering: We cannot delete or add words in the sentence. – Modeling: Models need to be specific on each word instead of sentiment classifi- cation on whole sentence. • Most of the words and phrases in sentences are not toxic. This indicates our dataset is imbalanced. The models we explore in this task include word- based Conditional Random Field (CRF), word- based Bidirectional Long Short Term Memory (Bi- LSTM) with and without pretrained word embed- ding, Bidirectional LSTM-CRF with pretrained word embedding. We choose Bi-LSTM-CRF as final …

Content truncated to 500 words