Rethinking Misinformation Detection: Accounting for Carey's Views of Communication

Abstract

When evaluated on social media datasets, misinformation detection systems are typically assessed as if social media postings are a homogeneous communicative genre. This thesis challenges that assumption by arguing that the strong presence of mass-media content in widely used datasets conceals models’ underperformance on real-world user-generated content. To address this issue, this study introduces a novel social media dataset with a labeling framework that distinguishes between news-generated and user-generated content. This allows for the first systematic comparison of language models’ misinformation detection performance across communicative genres. Model performance is analyzed using two generalized linear mixed models to investigate main effects and interactions related to content type, domain, prompting strategy, and model architecture. The results reveal a consistent performance gap in which models generally perform better on news-generated content than on user-generated content. However, the magnitude of this difference varies across domains and training approaches.