Garbage In, Garbage Out: Measuring and Predicting Insufficient Effort Responding in Survey Research

Abstract

The validity and reliability of conclusions drawn from survey data are contingent upon the quality of responses provided by participants. When analyzing survey data, researchers assume participants are engaged and exert a sufficient effort responding to the survey items. While it was initially believed that most survey research participants are attentive, a growing body of evidence suggests that up to 50% of participants (Francavilla et al., 2019; Meade & Craig, 2012) put forth an insufficient effort responding (IER). Although IER represents a significant threat to survey research integrity, important gaps persist in the literature pertaining to the detection and prediction of IER. On the one hand, a standardizable IER detection approach has yet to be developed. On the other hand, even if understanding the process underlying IER is crucial to prevent it, a robust and widespread examination of the predictors of IER remains to be conducted. This thesis sought to tackle the aforementioned gaps across six studies divided into three manuscripts.

In Manuscript 1, a novel standardizable IER detection approach (i.e., multidimensional insufficient effort responding detection approach; mIERda) was introduced and its criterion validity was examined across a series of four simulation studies. Building on Manuscript 1, Manuscript 2 relied on a prospective design to further the validation of the mIERda and assess its construct validity using 18 datasets (N = 5014). Lastly, drawing on 23 datasets (N = 2216) collected over the course of two years, Manuscript 3 examined the contextual (i.e., survey length, study topic, time of the semester) and personal (i.e., personality, global motivation, motivation for research participation) determinants of IER using a prospective longitudinal design. Taken together, findings arising from this thesis provide evidence of the criterion (Manuscript 1) and construct (Manuscript 2) validity of the mIERda. Results from Manuscript 3 demonstrate a limited contribution from contextual determinants and suggest that IER is mostly driven by personal determinants. Results also support that global autonomous motivation acts as a protective factor against IER at the start of the survey, whereas amotivation for research is tied to greater odds of IER throughout the survey. These findings provide meaningful insight to guide recommendations pertaining to IER detection, survey design, and motivational interventions with the overarching goal of lessening the prevalence of IER and its impact on survey research.