The need to evaluate research integrity when systematic reviewing: a Study Within A Review (SWAR)

Introduction

As two almost-doctors having just finished 6 years of medical school, the importance of evidence based medicine has been hammered into us. It is at the frontline of informing policy guidelines, advancing scientific knowledge, and empowering healthcare professionals and patients to make informed decisions based on reliable evidence. We therefore seized the opportunity to work with Dr David Nunan at the Centre for Evidence Based Medicine (CEBM) as part of our final year.  During our time with him we focused on research integrity and ultimately did a project assessing this.  In this report, we present the objectives of our study, the methodology employed, our progress so far, and insights gained.

What is the Problem?

Research misconduct is a topical issue, with issues of research integrity at its heart. The pivotal role of systematic reviews in informing health care practice, guidelines and policy as well as research priorities makes issues of research integrity of paramount importance. The assessment of research integrity within systematic reviews, however, is currently very limited and not systematically considered. A recent example demonstrating the importance of research integrity assessments in systematic reviews was the ivermectin trials for COVID-19, where initial systematic reviews and meta-analyses of early trials suggested a large beneficial effect. A subsequent critical appraisal of these trials showed methodological flaws and data irregularity which upon follow up revealed research misconduct and two studies being retracted along with at least one meta-analysis. In recognition of this issue, initiatives are underway to support systematic review authors to consider research integrity assessments. There are, however, few other examples of the application and impact of research integrity assessments in systematic reviews, and even fewer using available tools designed for such purposes.

What Did We Do?

In order to address this gap, our study will apply the Cochrane Pregnancy and Childbirth Trustworthiness Screening Tool (CPC-TST) to evaluate the impact of integrity assessments within an ongoing systematic review titled “The effects of presenting diagnostic accuracy and intervention efficacy statistics in different numerical formats: a systematic review”. 

We started by having several tutorials with Dr Nunan discussing EBM and research integrity more generally.  We talked about the use of modern artificial intelligence, such as ChatGPT, and how this could potentially be applied to facilitate the critical appraisal of papers.  We looked at a variety of methods to evaluate research integrity and ultimately settled on using the CPC-TST, as we felt it was easy to use and incorporated many of the essential aspects of research integrity such as retraction notices, ethical approval, prospective registration, similarity of baseline characteristics in randomized trials and can be applied to studies of non-pharmacological interventions (Figure 1).

We then drafted a protocol for our SWAR and came up with the following objectives:

To assess the effect of performing integrity assessments in a systematic review using the CPC-TST tool on review results, including:

• the inclusion and exclusion of studies within the review;
• pooled effect estimates and 95% confidence intervals;
• the overall certainty of evidence;
• the overall review conclusion(s)

To document the process of implementing research integrity checks (using the CPC-TST), including ease of use, limitations, impact on the systematic review process and areas for development.

We registered our protocol on the Open Science Framework and the Medical Research Council’s SWAR registry. 

Further Work

Although our allocated 3 weeks with Dr Nunan and the CEBM flew by, we both hope to further work on this project whilst starting our foundation training.  We hope to apply the CPC-TST to all eligible studies in the systematic review, document our findings for each criterion, and analyse the impact this has on the overall conclusions of the study.  We also hope to document the process of implementing this research integrity check to facilitate the refinement and standardization of integrity assessment methodologies.  We plan to publish our findings separately alongside the systematic review itself.

What Did We Learn?

Through our SWAR utilizing the CPC-TST, we gained valuable insights into the impact of integrity assessments on the systematic review process and outcomes. Dr Nunan’s takeaway lesson of “question everything” will remain with us for a long time to come.  We enhanced our understanding not only of EBM, its importance, and what it entails, but also of its flaws.  We learnt how important it is as doctors to be able to appraise the evidence for ourselves and how to do this.  We are very grateful to Dr Nunan and the CEBM for the opportunity to do this study module.