DES reproductions

Assessing the computational reproducibility of eight published healthcare DES models implemented in Python or R.

I worked on this from May 2024 to January 2025, with supervision and guidance Tom Monks. I also shared progress with Alison Harper, Nav Mustafee, and Andy Mayne.

Summary

A detailed study protocol was first developed, informed by existing reproducibility studies and pilot work. The workflow for assessing each study is summarised in the figure below.

The eight models were selected to ensure diversity across a range of factors including the health focus (e.g., healthcare condition, specific system), geographical context, and model complexity

Reproducing results required up to 28 hours of troubleshooting per model. Four models were judged to be fully reproduced, while four were partially reproduced - between 12.5% and 94.1% of reported outcomes.

Count, proportion, and time to reproduce items within the scope of each study. Inspired by a figure in Krafczyk et al. (2021).

Based on the barriers and facilitators observed during these reproductions, we developed the STARS reproducibility recommendations. These are presented in the figures below, divided into categories: recommendations that specifically support reproducibility, and those that were more relevant to troubleshooting models (and therefore also to reuse).

Recommendations to support reproducibility. Below each recommendation, a count of studies that fully met it is provided. The total may fall below eight if the criteria were not applicable to a given study (e.g., If they didn’t perform scenario analysis, or only provided one version of the code).

Recommendations to support troubleshooting and reuse. Below each recommendation, a count of studies that fully met it is provided. The total may fall below eight if the criteria were not applicable to a given study (e.g., If they didn’t have a web application, or didn’t have scenarios to vary parameters). Some recommendations were marked as “N/A” where it was not felt appropriate or feasible to count/assess their inclusion

Websites and GitHub repositories

We have described this work in a publication in the Journal of Simulation.

The work is also documented in a dedicated Quarto summary website, which provides more fine-grained detail on the reproductions.

Each of the eight DES models has its own research compendium-style GitHub repository, with a corresponding website and archival record, as linked in the table below. Each repository was created from a template which we developed during pilot work. In pilot work, an example model from colleagues was used to test and refine the reproducibility protocol, resulting in the repository stars-reproduce-allen-2020.

Reproduction study	Website	GitHub	Zenodo
Shoaib and Ramamohan 2022	Website	stars-reproduce-shoaib-2022	10.1177/00375497211030931
Huang et al. 2019	Website	stars-reproduce-huang-2019	10.5281/zenodo.12657280
Lim et al. 2020	Website	stars-reproduce-lim-2020	10.5281/zenodo.12795365
Kim et al. 2021	Website	stars-reproduce-kim-2021	10.5281/zenodo.13121136
Anagnostou et al. 2022	Website	stars-reproduce-anagnostou-2022	10.5281/zenodo.13306159
Johnson et al. 2021	Website	stars-reproduce-johnson-2021	10.5281/zenodo.13832333
Hernandez et al. 2015	Website	stars-reproduce-hernandez-2015	10.5281/zenodo.13832260
Wood et al. 2021	Website	stars-reproduce-wood-2021	10.5281/zenodo.13881986

GW4 Open Research Prize - Improving Quality

This research was shortlisted for the “Improving Quality” prize, which is “for those able to demonstrate that the quality of their research has been enhanced through the adoption of open research practices in their work”. You can find out more about the prize and the other shortlisted entries and winners here.

Amy Heather presented this work at the GW4 Open Research Prize awards event. The slides were created using Quarto: see the slides and accompanying slides GitHub repository. A recording of the presentation is available:

STARS

This work was completed as part of the project STARS: Sharing Tools and Artefacts for Reproducible & Reusable Simulations in healthcare. I have created a website where you can find out more about this project: https://pythonhealthdatascience.github.io/stars/.