Are systematic reviews as trustworthy as we think? A case for critical appraisal bias in back pain research.

Introduction

As final year medical students, the importance of using high-quality research to support clinical decision-making has been emphasised to us repeatedly over the course of our degree. In order to do this successfully, it is vital to be able to recognise and evaluate the potential of bias. Bias is a large topic within evidence based medicine (EBM) due to its numerous forms and far-reaching consequences, making it a challenging yet important area to understand. We are grateful to have been able to join David at the CEBM to improve our abilities in assessing bias, particularly in systematic reviews. We were also supported by one of David’s current EBHC DPhil students, Christopher Banks-Pillar.

What’s the problem we addressed?

Implicit bias is known to affect healthcare delivery and outcomes. One potential form of implicit bias in healthcare is the bias arising from negative attitudes towards those with conditions that are associated with modifiable risk factors, and which are seen as treatable with non-pharmacological interventions involving  behavioural or lifestyle modification. Examples of such conditions include obesity, type 2 diabetes and non-specific chronic pain. Studies have explored clinicians’ and the public’s views of personal health responsibility and raise concerns that these attitudes may play an important role in health policy. For example, in the treatment of conditions associated with modifiable risk factors, pro-effort behavioural interventions may be valued as morally positive compared with pharmacological interventions.

Given the existence of these attitudes in clinicians and the public, it is possible that healthcare researchers also exhibit similar biases. This would be concerning in the case of systematic review authors as these outputs often guide health policy by evaluating the quality of evidence supporting different interventions. Little has been studied about the potential role of implicit biases in the conduct of systematic reviews assessing treatments for conditions with modifiable risk factors. We focused on a specific element of systematic reviews most susceptible to implicit biases due to it requiring a degree of subjective judgement – the risk of bias assessment, which assesses the validity of a study’s findings.

What did we do?

Before starting on this project, we first needed to gain relevant background knowledge, so the first tutorial with David was spent reviewing the different types of bias and understanding how systematic reviews assess risk of bias for their included trials. We then spent some time getting to grips with conducting risk of bias assessments ourselves using the Cochrane risk of bias (RoB) 2.0 tool.

During the learning process, we simultaneously searched the literature for studies describing the form of bias we wanted to investigate among healthcare researchers. This yielded few related studies, which was both unsettling, as it meant this form of bias might not exist amongst healthcare researchers or isn’t considered important enough to study. It was equally exciting because it meant we might be investigating something that is important but has had little attention.

After deciding to go ahead with the idea, we began working on the protocol for an audit of systematic reviews to evaluate the performance of risk of bias assessments comparing treatments for conditions with modifiable risk factors, using lower back pain as a case study. We plan to publish our protocol and begin data collection with help from David and Chris soon.

What did we learn?

The Special Studies Module in EBM has reinforced to us the methodology behind conducting systematic reviews. Importantly, it has also emphasised to us the need to think critically about all forms of medical research, particularly types of publications that are often assumed to be of better quality merely due to their elevated position in the ‘hierarchy of evidence’. By learning about different risk of bias assessment tools, including the latest Cochrane RoB2 tool, we have been able to appreciate their strengths and weaknesses, and further consider how deficits in their use might impact clinical research conclusions. Our time with CEBM has shown us the complexity of EBM and been a fantastic opportunity to improve vital skills for our future careers.