Can we trust the integrity of trial evidence informing UK clinical guidelines?

What did we do?

We started with an introductory meeting with David, to discuss our individual goals for this SSM. We were joined by Dr Jeffrey Aronson who is supporting the project. We later reviewed some of the principles of evidence-based medicine (EBM), consolidating and expanding some of the EBM knowledge we had covered in our fourth year as medical students. We particularly focussed on research integrity, which we later explored in our project.

Research integrity refers to the correct and accurate reporting of scientific methods and results;that is, the degree to which we can trust that research has been carried out properly, as described by authors. Irregularities in integrity may relate to ethical approval and simple human errors in reporting, such as incorrectly typed data, or to insufficiently detailed explanations of randomisation procedures; however, they can also include more serious and intentional infractions, including falsified data and so called “zombie” trials, trials that are fatally flawed due to containing false data.

Our project will focus on applying tools that attempt to flag studies with potential intergrity issues. We plan to apply these tools to randomised controlled trials (RCTs) that assessed pharmacological and non-pharmacological interventions and were cited in NICE clinical practice guidelines.

What did we find?

Two tools have been developed to help individuals assess research integrity: the Research Integrity Assessment (RIA) tool and the Cochrane Pregnancy and Childbirth Trustworthiness Screening Tool (CPC-TST).  Our initial attempts to use these tools were moderately hampered by a lack of clear guidance on their use. With much left for interpretation as we designed our protocol for our study, we discussed between ourselves and with colleagues the ways in which we should apply these tools. For example, we noted that many older studies were likely to raise a flag based on the tools, in part because of reporting norms from the time they were published. Simply stating that a study was randomised was at one time considered sufficient, while now details of randomisation should be elaborated (e.g. use of block randomisation vs “true” computer randomisation).

 We also found that many aspects of the appraised research warranted clarification by the authors before a final decision could be made as to the level of integrity. For example, apparent data errors may be explained away reasonably, or methods can be clarified beyond what is written in a published journal article.

Implications and discussion

Research integrity has long been an area of interest, championed by researchers such as Professor Ben Mol and Dr Elizabeth Bik. However, in the wake of the covid-19 pandemic it has been thrust into centre-stage, particularly in the context of ivermectin as a discredited treatment. Raising the profile of research integrity concerns is needed, to build trust in the scientific process.

Prof Mol suggests that up to 20% of research may have research integrity issues. This raises the  question “What is the reliabilility and trustworthiness of evidence underpinning recommendations in current clinical practice guidelines?”

The aim of our project is to assess the extent to which RCTs, evidence from which underpins UK clinical practice guideline recommendations, create doubts about research integrity, according to criteria set out in the RIA tool and CPC-TST.

What did we learn and where next?

We have really enjoyed our time with EBM and learnt valuable skills in critical appraisal, as well as designing and publishing study protocols. This will provide us with a strong basis for our future careers. We hope to continue this project and eventually submit the results for publication.

 Minahil, Oskar, and Sukhmunni