In this blogpost, we highlight the following study on a chat tool designed to reduce political polarization as a potential candidate for a robustness report:
Combs, A., Tierney, G., Guay, B. et al. Reducing political polarization in the United States with a mobile chat platform. Nature Human Behaviour, 7, 1454–1461 (2023). https://doi.org/10.1038/s41562-023-01655-0
The findings of Combs et al. (2023) suggest that anonymous online cross-party conversations about contentious issues (i.e., immigration or gun control) can help depolarize the public. Specifically, it appears to identify an effective paradigm for studying and mitigating political polarization; basically, a chat tool to reduce political polarization! Consequently, in recent years, it has received considerable attention within the polarization literature as experimental evidence that engaging in political discussions with individuals holding opposing partisan identities or issue positions in online settings can yield beneficial outcomes.
While other experimental studies likewise support the notion that online discourse can promote depolarization (e.g., 1,2,3), the overall number of such experiments remains relatively small, and replications are rare. Given widespread concerns about polarization, including detrimental effects on health4, it seems essential to evaluate the robustness of the findings; does this chat tool indeed hold a (small) key to political moderation, or is the sceptic’s reflex justified that it might simply be too good to be true?
Here is the abstract:
Do anonymous online conversations between people with different political views exacerbate or mitigate partisan polarization? We created a mobile chat platform to study the impact of such discussions. Our study recruited Republicans and Democrats in the United States to complete a survey about their political views. We later randomized them into treatment conditions where they were offered financial incentives to use our platform to discuss a contentious policy issue with an opposing partisan. We found that people who engage in anonymous cross-party conversations about political topics exhibit substantial decreases in polarization compared with a placebo group that wrote an essay using the same conversation prompts. Moreover, these depolarizing effects were correlated with the civility of dialogue between study participants. Our findings demonstrate the potential for well-designed social media platforms to mitigate political polarization and underscore the need for a flexible platform for scientific research on social media.
We suggest this article as a promising target for a robustness analysis, given its considerable policy influence and – at least at first glance – somewhat fragile evidence. The authors did conduct several checks, including Bayesian additive regression trees, but their findings could benefit from an independent dissection. To offer a sneak peek, a quick Bayesian t-test on the depolarization index gives a Bayes factor of BF₊₀ = 1.35, which offers only anecdotal evidence, too weak to clearly favor either the null or a treatment effect.
Potential dimensions for robustness exploration include how polarization is defined, the role of covariates, and modeling choices. Combs et al. (2023) combined 21 indicators into a global polarization index and two subscales: issue (ideological) polarization and affective polarization (e.g., warmth toward one’s own party and hostility toward the other). Yet alternative definitions are possible, such as focusing on social distance or stereotyping. Other analytic decisions may likewise shape the results, such as how missing data are handled, which covariates are included (e.g., region, age, education, political interest), and what statistical models are used. For example, multilevel models could help account for regional differences.
Author
Alexandra Sarafoglou & Suzanne Hoogeveen
References
Combs, A., Tierney, G., Guay, B. et al. Reducing political polarization in the United States with a mobile chat platform. Nature Human Behaviour, 7, 1454–1461 (2023). https://doi.org/10.1038/s41562-023-01655-0
1. Kalla, J. L., & Broockman, D. E. (2020). Reducing Exclusionary Attitudes through Interpersonal Conversation: Evidence from Three Field Experiments. American Political Science Review, 114 (2), 410–425. https://doi.org10.1017/S0003055419000923
2. Fishkin, J., Siu, A., Diamond, L., & Bradburn, N. (2021). Is Deliberation an Antidote to Extreme Partisan Polarization? Reflections on “America in One Room.” American Political Science Review, 115 (4), 1464–1481. https://doi.org/10.1017/S0003055421000642
3. Hobolt, S. B., Lawall, K., & Tilley, J. (2024). The Polarizing Effect of Partisan Echo Chambers. American Political Science Review, 118(3), 1464–1479. https://doi.org/10.1017/S0003055423001211
4. Van Bavel, J.J., Gadarian, S.K., Knowles, E., & Ruggeri, K. Political polarization and health. Nature Medicine, 30, 3085–3093 (2024). https://doi.org/10.1038/s41591-024-03307-w