In this blogpost we wish to highlight the following article as a candidate for a Robustness Report:
Stagnaro, M. N., Druckman, J., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. G. (2026). Representativeness and response validity across nine Opt-In online samples. Nature Human Behaviour. https://doi.org/10.1038/s41562-026-02438-z
The abstract:
The study of human behaviour has shifted in the past 15 years. It increasingly relies on opt-in non-probability online data sources. Here we offer an analysis of nine such sources (N = 13,053), aiming to inform researchers conducting experiments or correlational studies. We assess response validity (attentiveness, effort, honesty, speeding and attrition), the extent to which samples represent the underlying population (demographics, attitude representativeness and experimental treatment effects) and professionalism (number and frequency of studies taken and modality of device on which studies are taken). We document substantial variation across samples on each dimension. Samples using demographic quotas display relatively higher amounts of representativeness across multiple indicators (beyond demographics) but often exhibit less response validity. However, inclusion of two attention checks early in a survey can enhance response validity without negatively impacting representativeness. We offer guidance for choosing opt-in samples, depending on the purpose of the research and resource constraints.
We suggest this target article for reanalyses because of the impact these findings can have on the data quality of future opt-in surveys. The authors reported that different platforms vary on response validity and representativeness, and that there tends to be a trade-off between the two. Additionally, “professional” participants (i.e., completed many surveys, mainly on desktop) have higher response validity but tend to represent a smaller portion of the population. A reanalysis could strengthen our confidence in the findings and help inform future research on what survey platforms are appropriate for the scope of a specific study.
The data are publicly available at https://osf.io/2jq94/overview?view_only=941b1e0605b441acbcf8112944f4b13a. To express your interest in re-analysing the data from the original article, please fill out this sheet.
Author
Jari Holla
References
Stagnaro, M. N., Druckman, J., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. G. (2026). Representativeness and response validity across nine Opt-In online samples. Nature Human Behaviour. https://doi.org/10.1038/s41562-026-02438-z
Featured image from GenAI