Social Fairness in Semi-Supervised Toxicity Text Classification

Abstract

The rapid growth of user-generated content on social media platforms in the form of text caused moderating toxic language manually to become an increasingly challenging task. Consequently, researchers have turned to artificial intelligence (AI) and machine learning (ML) models to detect and classify toxic comments automatically. However, these models often exhibit unintended bias against comments containing sensitive terms related to de- mographic groups, such as race and gender, which leads to unfair classifications of samples. In addition, most existing research on this topic focuses on fully supervised learning frame- works. Therefore, there is a growing need to explore fairness in semi-supervised toxicity detection due to the difficulty of annotating large amounts of data. In this thesis, we aim to address this gap by developing a fair generative-based semi-supervised framework for mitigating social bias in toxicity text classification. This framework consists of two parts, first, we trained a semi-supervised generative-based text classification model on the bench- mark toxicity datasets. Then, in the second step, we mitigated social bias in the trained classifier in step 1 using adversarial debiasing, to improve fairness. In this work, we use two different semi-supervised generative-based text classification models, NDAGAN and GANBERT (the difference between them is that the former adds negative data augmenta- tion to address some of the problems in GANBERT), to propose two fair semi-supervised models called FairNDAGAN and FairGANBERT. Finally, we compare the performance of the proposed fair semi-supervised models in terms of accuracy and fairness (equalized odds difference) against baselines to clarify the challenges of social fairness in semi-supervised toxicity text classification for the first time. Based on the experimental results, the key contributions of this research are: first, we propose a novel fair semi-supervised generative-based framework for fair toxicity text classification for the first time. Second, we show that we can achieve fairness in semi- supervised toxicity text classification without considerable loss of accuracy. Third, we demonstrate that achieving fairness at the coarse-grained level improves fairness at the fine-grained level but does not always guarantee it. Fourth, we justify the impact of the labeled and unlabeled data in terms of fairness and accuracy in the studied semi- supervised framework. Finally, we demonstrate the susceptibility of the supervised and semi-supervised models against data imbalance in terms of accuracy and fairness.