Why COVID-19 modelling of progression and prevention fails to translate to the real-world

Mathematical models were used widely to inform policy during the COVID pandemic. However, there is a poor understanding of their limitations and how they influence decision-making. We used systematic review search methods to find early modelling studies that determined the reproduction number and analysed its use and application to interventions and policy in the UK. Up to March 2020, we found 42 reproduction number estimates (39 based on Chinese data: R0 range 2.1–6.47). Several biases affect the quality of modelling studies that are infrequently discussed, and many factors contribute to significant differences in the results of individual studies that go beyond chance. The sources of effect estimates incorporated into mathematical models are unclear. There is often a lack of a relationship between transmission estimates and the timing of imposed restrictions, which is further affected by the lag in reporting. Modelling studies lack basic evidence-based methods that aid their quality assessment, reporting and critical appraisal. If used judiciously, models may be helpful, especially if they openly present the uncertainties and use sensitivity analyses extensively, which need to consider and explicitly discuss the limitations of the evidence. However, until the methodological and ethical issues are resolved, predictive models should be used cautiously.