On the Interaction between Annotation Quality and Classifier Performance in Abusive Language Detection

Proceedings of Recent Advances in Natural Language Processing, pages 73–78 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_009 73 Enriching the Transformer with Linguistic Factors for Low-Resource Machine Translation Jordi Armengol-Estap´e, Marta R. Costa-juss`a, Carlos Escolano TALP Research Center, Universitat Polit`ecnica de Catalunya, Barcelona jordi.armengol.estape@gmail.com,{marta.ruiz,carlos.escolano}@upc.edu Abstract Introducing factors, that is to say, word fea- tures such as linguistic information referring to the source tokens, is known to improve the re- sults of neural machine translation systems in certain settings, typically in recurrent architec- tures. This study proposes enhancing the cur- rent state-of-the-art neural machine translation architecture, the Transformer, so that it allows to introduce external knowledge. In particu- lar, our proposed modification, the Factored Transformer, uses linguistic factors that insert additional knowledge into the machine transla- tion system. Apart from using different kinds of features, we study the effect of different architectural configurations. Specifically, we analyze the performance of combining words and features at the embedding level or at the encoder level, and we experiment with two different combination strategies. With the best-found configuration, we show improve- ments of 0.8 BLEU over the baseline Trans- former in the IWSLT German-to-English task. Moreover, we experiment with the more chal- lenging FLoRes English-to-Nepali benchmark, which includes both extremely low-resourced and very distant languages, and obtain an im- provement of 1.2 BLEU. 1 Introduction Many classical Natural Language Processing (NLP) pipelines used linguistic features (Koehn and Hoang, 2007; Du et al., 2016). In recent years, the rise of neural architectures has dimin- ished the importance of the aforementioned fea- tures. Nevertheless, some works have still shown the effectiveness of introducing linguistic informa- tion into neural machine translation systems, typi- cally in recurrent sequence-to-sequence (Seq2seq) architectures (Sennrich and Haddow, 2016; Garc´ıa- Mart´ınez et al., 2016; Espa˜na-Bonet and van Gen- abith, 2018). By factored Neural Machine Trans- lation (NMT), we refer to the use of word features alongside the words themselves to improve transla- tion quality. Both the encoder and the decoder of a Seq2seq architecture can be modified to obtain bet- ter translations (Garc´ıa-Mart´ınez et al., 2016). The most prominent approach consists of modifying the encoder such that instead of only one embedding layer, the encoder has as many embedding layers as factors, one for words themselves and one for each feature, and then the embedding vectors are concatenated and input to the rest of the model, which remains unchanged (Sennrich and Haddow, 2016). The embedding sizes are set according to the respective vocabularies of the features. Note that they used Byte Pair Encoding (BPE) (Sennrich et al., 2016), an unsupervised preprocessing step for automatically splitting words into subwords with the goal of improving the translation of rare or unseen words. Thus, the features had to be repeated for each subword. In Espa˜na-Bonet and van Genabith (2018), the exact same architecture was used, except that this new proposal used concepts extracted from linked data database, BabelNet (Navigli and Ponzetto, 2012). These semantic features, synsets, were shown to improve zero-shot translations. All the cited works obtained moderate improvements with …

Content truncated to 500 words