Numbers don't count - methods do
Why we need to reshape critical appraisal and develop tools that allow differentiated evaluations of the myriad of qualitative methodological approaches
20 years have passed since David Sackett and John Wennberg said we need to stop ‘squabbling over the best methods’. Yet, there are still plenty of arguments about best methods, so why are we still ignoring their plea? So we will try and focus on the questions to be asked and make improvements to the debate. And here’s why.
The majority of evidence underpinning evidence-based medicine (EBM) still largely represents a positivist perspective (i.e. quantitative methods) of generating and applying knowledge in health care practice. Research based on an interpretivist perspective (i.e. qualitative methods) plays second fiddle – being left out or resigned to the bottom of most EBM evidence hierarchies.
In the first of two articles published in the BMJ EBM learning series, we argue why qualitative research should be an integral part of EBM as it provides important evidence on questions around ‘how’ and ‘why’ that often can’t be answered by a yes/no quantitative approach. In addition, we also stress the importance of asking ‘why’ in clinical practice, a question it appears isn’t asked all that often.
Qualitative research designs fall within an interpretivist/ constructivist paradigm, which cannot, and indeed does not aim to make definitive statements about research findings. Rather the aim is to explore different perspectives and experiences of phenomena rather than assessing associations and differences using statistical methods. The underlying premise of qualitative research does not assume one verifiable or replicable truth, but several ‘truths’ or perspectives that are equally valid. Thus, research within this paradigm does not propose to falsify a hypotheses with degrees of certainty. As such, the former is less likely to produce ‘headline grabbing’ results compared to the latter (observational cohort studies of nutrition for example).
A fundamental element of EBM practice is scrutiny of evidence for its rigour and application to practice. Despite a large body of literature focusing on appraisal and rigour of quantitative research (often referred to as ‘trustworthiness’ in qualitative research), there remains debate about how to – and even whether to – critically appraise qualitative research. In a second paper published in the BMJ EBM, we present our case that current approaches to critical appraisal of qualitative research designs are not fit for purpose and propose solutions.
Appraising the quality of qualitative research is not a new concept – there are a number of published appraisal tools, frameworks and checklists in existence. The main issue concerning these appraisal methods is that they take a broad-brush approach to ‘qualitative’ research as a whole, with few, if any, sufficiently differentiating between the different methodological approaches. It’s akin to saying “here’s a tool for assessing the quality of ALL quantitative studies.”
Moreover, an important and often overlooked point is the distinction between tools designed for appraising methodological quality and tools designed for assessing the quality of methods reporting. An example is the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist, which was designed to provide standards for authors when reporting qualitative research but is often mistaken for a methods appraisal tool.
As qualitative research becomes ever more embedded in health science research, and in order for that research to better impact on health care decisions, we need to reshape critical appraisal and develop tools that allow differentiated evaluations of the myriad of qualitative methodological approaches rather than continuing to treat qualitative research as a single unified approach.
Veronika Williams and Anne-Marie Boylan are Senior Researchers at the Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford. They lead and tutor on introductory and advanced level qualitative research courses as part of the MSc in Evidence-Based Health Care.
David Nunan is a Senior Researcher and Director of Postgraduate Certificate in Teaching Evidence-Based Health Care. He is based at the Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford.