In the previous blogpost we highlighted the following article as a candidate for a set of Robustness Reports:
Zhai, X., Tong, H. H. Y., Lam, C. K., Xing, A., Sha, Y., Luo, G., Meng, W., Li, J., Zhou, M., Huang, Y., Wong, L. S., Wang, C., & Li, K. (2024). Association and causal mediation between marital status and depression in seven countries. Nature Human Behaviour, 8(12), 2392–2405. https://doi.org/10.1038/s41562-024-02033-0
We also stated that the datasets used are publicly available and can be found in the paper’s data availability section. On closer inspection, however, this does not appear to be correct. The data availability section reads:
Data from the NHANES are available at https://www.cdc.gov/nchs/nhanes/index.htm. Data from the WLS are available on the application at https://wls.wisc.edu/. Data from the APMS are available on the application at https://ukdataservice.ac.uk/find-data/. Data from the MHAS are available on the application at https://www.mhasweb.org/Home/index.aspx. Data from KLoSA are available on the application at https://survey.keis.or.kr/eng/klosa/klosa01.jsp. Data from TILDA are available on the application at https://tilda.tcd.ie/. Data from CHARLS are available on the application at https://charls.pku.edu.cn/. Data from the IFLS are available on the application at https://www.rand.org/well-being/social-and-behavioral-policy/data/FLS/IFLS.html.
These are generic websites, and this is not really helpful for the researcher who wishes to execute an alternative analyses on the authors’ data. There is also a code availability section, and this reads:
The computational code for causal mediation analysis with bootstrap resampling is shared at https://github.com/Li-OmicsLab/Causal_NHB.
Indeed, the GitHub page has R code, but lacks the crucial data file; in the screenshot below, “unmarried.sav”.
When I first noticed this, I became somewhat upset (“what can possibly constitute an acceptable reason to withhold the .sav file after proper anonymization?”), and even started to think about contacting the authors (“why did you withhold the .sav file?”) and the editors (“why does NHB not check these data availability statements more carefully?”). After calming down, I realized that this secrecy may not be the authors’ fault. I am told that these database websites simply prohibit their users from sharing data publicly, and this means there is a legal barrier. This is deeply unfortunate of course, and an impediment to scientific progress. [NB. the work does contain a host of private information, but it does look as if most, if not all of it, could be properly anonymized].
More information on the Journal of Robustness Reports is available on this site and on BayesianSpectacles.
Author
EJ Wagenmakers
References
Zhai, X., Tong, H. H. Y., Lam, C. K., Xing, A., Sha, Y., Luo, G., Meng, W., Li, J., Zhou, M., Huang, Y., Wong, L. S., Wang, C., & Li, K. (2024). Association and causal mediation between marital status and depression in seven countries. Nature Human Behaviour, 8(12), 2392–2405. https://doi.org/10.1038/s41562-024-02033-0